Source-linked AI summary
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, Xiang Bai
TL;DR
LMMs struggle with high-resolution input and detailed scene understanding, especially when training captions are sparse. Monkey addresses this with patch-based resolution enhancement and multi-level description generation, achieving strong results across multimodal benchmarks, while remaining limited to six patches and bounded description knowledge.
Problem
LMMs face challenges handling larger image resolutions and linking detailed visual features to the short captions common in existing datasets.
Method
Monkey uses sliding-window image patches with adapted shared encoders and global features, alongside automatically generated multi-level descriptions from multiple specialized systems.
Results
Monkey surpasses Qwen-VL in most document-oriented VQA tasks, with an average improvement of 9.77%, and reports strong performance across multiple benchmarks.
Takeaways & Limitations
The combined designs provide a training-efficient way to improve high-resolution understanding and contextual scene-object associations in LMMs.
Takeaways & Limitations
Input processing is limited to six patches, restricting further resolution expansion; description generation is bounded by BLIP2 and original CC3M world knowledge.
Abstract
from arXiv · showhide
Large Multimodal Models (LMMs) have shown promise in vision-language tasks but struggle with high-resolution input and detailed scene understanding. Addressing these challenges, we introduce Monkey to enhance LMM capabilities. Firstly, Monkey processes input images by dividing them into uniform patches, each matching the size (e.g., 448x448) used in the original training of the well-trained vision encoder. Equipped with individual adapter for each patch, Monkey can handle higher resolutions up to 1344x896 pixels, enabling the detailed capture of complex visual information. Secondly, it employs a multi-level description generation method, enriching the context for scene-object associations. This two-part strategy ensures more effective learning from generated data: the higher resolution allows for a more detailed capture of visuals, which in turn enhances the effectiveness of comprehensive descriptions. Extensive ablative results validate the effectiveness of our designs. Additionally, experiments on 18 datasets further demonstrate that Monkey surpasses existing LMMs in many tasks like Image Captioning and various Visual Question Answering formats. Specially, in qualitative tests focused on dense text question answering, Monkey has exhibited encouraging results compared with GPT4V. Code is available at https://github.com/Yuliang-Liu/Monkey.
1. Introduction
Monkey addresses LMM limitations in high-resolution visual processing and sparse image descriptions through patch-based resolution enhancement and multi-level caption generation. The approach supports broader multimodal evaluation, including dense-text understanding.
- LMMs benefit from high-resolution images because finer visual details support object, relationship, context, and detailed-caption understanding.
- Existing approaches increase resolution through larger pretrained visual modules or curriculum learning, but require substantial resources and still face large-image challenges.
- Monkey divides high-resolution images into smaller patches processed by adapted static visual encoders, increasing resolution without extensive pretraining.
- Monkey’s multi-level description generation combines multiple specialized systems to produce abundant, layered captions covering visual and textual details.
- Across 18 datasets, Monkey reports competitive performance in image captioning, general VQA, scene-text VQA, and document-oriented VQA, with promising dense-text QA results versus GPT4V.
2. Related Work
Related work has advanced multimodal representation, vision-language alignment, broad task training, and structured visual answering. However, extracting finer image features remains an open challenge.
- Prior LMMs integrate vision encoders with resamplers, Q-Formers, or multimodal representations to connect images and language.
- Later systems improve image-text alignment, modality collaboration, positional outputs, and task-specific visual answering across varied applications.
- Despite these advances, prior work identifies continued challenges in extracting finer image features.
3. Methods
Monkey combines patch-based high-resolution processing, global-local feature integration, multi-level descriptions, and multi-task training. Its design targets the mismatch between detailed images and simple training captions.
- 3.1. Enhancing Input Resolution: Monkey segments images into patches processed by shared ViT components with distinct adapters, then combines local and global features with the question for LLM answering.
- 3.1. Enhancing Input Resolution: Sliding-window partitioning and LoRA help shared encoders capture detail-sensitive features across image regions without substantially increasing parameters or computation.
- 3.1. Enhancing Input Resolution: The method preserves a resized global image while processing local patches concurrently, balancing detailed and holistic visual information without substantial computational growth.
- 3.2. Multi-level Description Generation: Simple image-text pairs in common datasets often lack the detail needed to align high-resolution visual features with language.
- 3.2. Multi-level Description Generation: Monkey generates layered descriptions by combining global captions, region descriptions, OCR text, object parts, spatial information, consistency filtering, and ChatGPT refinement.
- 3.2. Multi-level Description Generation: The resulting caption process captures broad visual and textual nuances in a comprehensive and contextually diverse format.
4. Experiment
Monkey is evaluated across image captioning, general, scene-text, and document-oriented VQA tasks, with ablations examining resolution, adapters, and multi-level descriptions. Results show broad gains, especially on text- and document-centric benchmarks, alongside qualitative advantages in detailed descriptions and complex text-based inquiries.
- Image Captioning: Monkey improves image captioning performance on Flickr30K and TextCaps while producing more detailed image descriptions than GPT4V in qualitative comparisons.Monkey identifies scene elements such as store names, people, clothing, and accessories in the detailed-caption comparison.
- General VQA: 1.62%: Monkey surpasses the nearest competing method by an average of 1.62% across four reported general VQA benchmarks.The reported benchmarks are VQAv2, OKVQA, ScienceQA, and VizWiz.
- Scene Text-centric VQA: 4.37%: Monkey outperforms the nearest competitor by an average of 4.37% across TextVQA, AI2D, STVQA, and ESTVQA.The authors associate the improvement with increased resolution and detailed caption data that provide clearer text and additional textual context.
- Document-oriented VQA: 9.77%: Monkey improves over Qwen-VL by an average of 9.77% across document-oriented VQA tasks.The evaluated tasks include DocVQA, ChartQA, InfographicVQA, DeepForm, KLC, and WTQ.
- Resolution and Adapter Ablations: Increasing input resolution generally improves performance, with further gains at 1344×896 on DeepForm, InfoVQA, and WTQ but a slight TextVQA decline.Increasing resolution from 224 to 448 also yields significant improvements on LLaVA1.5; one or four LoRAs have minimal throughput differences.
- Resolution and Adapter Ablations: Four LoRA modules outperform fewer adapters, while replacing CC3M annotations with generated multi-level descriptions consistently improves GQA, TextVQA, and MMVet.Using one LoRA improves local-detail perception over no LoRA, especially on STVQA.
- Limitations: The method is limited to six patches because of the language model’s input length, and its description generation cannot identify a photographed location’s country beyond visual scene details.These constraints limit further input-resolution expansion and the scope of world knowledge expressed in generated descriptions.
5. Conclusion
Monkey combines training-efficient high-resolution input with multi-level descriptions to improve contextual scene understanding. It achieves strong results across multiple benchmarks, including promising dense-text performance compared with GPT4V.
- Monkey increases input resolution capacity to 1344×896 pixels without pretraining from the start.
- Its multi-level description generation method provides rich information for learning contextual associations between scenes and objects.
- Monkey achieves excellent results on multiple benchmarks, with promising image-captioning and dense-text document-image performance compared with various LMMs.
A. Summary of the Evaluation Benchmarks.
The evaluation spans image captioning, general VQA, scene-text-centric VQA, and document-oriented VQA benchmarks. These datasets cover natural images, text, diagrams, science topics, visual reasoning, and document understanding.
- Image Caption: The benchmark suite includes Flickr30K and TextCaps for image captioning, covering natural images with and without text.
- General VQA: General VQA evaluation uses VQAV2, OKVQA, GQA, ScienceQA, and VizWiz for open-ended, knowledge-based, compositional, science, and accessibility-focused questions.
- Scene Text-centric VQA: Scene Text-centric VQA includes TextVQA, AI2Diagram, STVQA, and ESTVQA, emphasizing reading and reasoning about text in images and diagrams.
- The evaluation overview and corresponding metrics are summarized in Table 7.
B. More Visualization Results
Additional visualizations show Monkey handling diverse VQA and captioning cases. The examples emphasize fine text perception, scene reasoning, chart understanding, and captioning ability.
- VQA: Monkey identifies image elements relevant to VQA questions and perceives minute text within images.
- VQA: Monkey reasons about objects in scenes and demonstrates strong understanding of visual charts.
- Captioning: The visualizations also showcase Monkey’s captioning ability.
C. More Examples of our Generated Data
Generated-data examples compare Monkey’s detailed captions and task performance with other multimodal models. The examples highlight broader image coverage, text sensitivity, and strong dense-text VQA results.
- Generated Captions: Monkey’s generated descriptions cover substantially more image details than the original CC3M annotations.
- VQA Comparisons: On dense-text VQA tasks, Monkey surpasses QwenVL-Chat, LLaVA-1.5, and mPLUG-Owl2 and shows promising results compared with GPT-4V.
- Detailed Caption Comparisons: Compared with GPT-4V, QwenVL-Chat, LLaVA-1.5, and mPLUG-Owl2, Monkey accurately describes images while remaining sensitive to embedded text.
E. Visualization results for models at different resolutions.
Visualization tests show that higher image resolution preserves visual details and improves VQA correctness. At 896 resolution, results were correct, whereas lower resolutions produced increasingly more errors and less visible detail.
- 896 resolution produced correct VQA results, while 784 and 672 produced errors, with 672 showing more errors.
- The VQA visual results further validate the importance of the size expansion method for improving LMM performance.
- As resolution decreased from 896 to 784 and 672, image details became progressively less visible to the model.
- The detailed-caption visualization marks accurately described text in red and inaccurately described text in blue.
F. Data Generation.
The data-generation pipeline uses empirically selected hyperparameters and autonomous detailed-caption generation. BLIP2 was chosen for large-scale description generation because it required substantially less time than the other compared VLMs.
- SAM’s default threshold and a 0.5 Image-Text Matching Score were empirically selected as effective hyperparameters.Quantitative validation used 80 samples evaluated with GPT-4V; the SAM threshold was relatively robust, while 0.5 yielded better performance.
- Table 8 reports the hyperparameter-control experiments, and Table 9 reports performance for different LMMs.
- Unlike LLaVA’s GPT4 method, the approach generates original detailed captions autonomously rather than relying on manually annotated COCO captions.
- BLIP2 required approximately 3 days to generate 5M descriptions, compared with about 21 days for LLaVA and 32 days for mPLUG-Owl.GPT-4V evaluation showed similar performance for brief local-area descriptions from the compared VLMs.
G. Ablation study on Global Feature.
Adding global features at 896 resolution improved performance on TextVQA, GQA, and DocVQA, supporting their contribution to overall performance.
- Adding global features at 896 resolution yielded gains of 7.5% on TextVQA, 0.6% on GQA, and 6.2% on DocVQA.