LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Yanzhe Zhang; Ruiyi Zhang; Jiuxiang Gu; Yufan Zhou; Nedim Lipka; Diyi Yang; Tong Sun

Source-linked AI summary

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tong Sun

arXiv:2306.17107v2Jun 29, 2023cs.CVcs.CL

TL;DR

Visual instruction-tuned models struggle to understand text within images. LLaVAR augments LLaVA with OCR-derived noisy data and GPT-4-generated conversations from text-rich images, substantially improving text-based understanding while slightly improving natural-image performance.

Problem
Visual instruction-tuned models struggle with textual details in images, although textual understanding is integral to everyday visual perception.
Method
LLaVAR uses 422K OCR-based noisy examples and 16K GPT-4-generated conversations from text-rich LAION images to augment LLaVA’s pretraining and fine-tuning.
Results
LLaVAR significantly improves text understanding within images while slightly improving performance on natural images across benchmark and instruction-following evaluations.
Takeaways & Limitations
The enhanced capability supports end-to-end interactions with online content that combines text and images.
Takeaways & Limitations
The collected instruction-following data is less diverse and substantial than LLaVA’s, partly because GPT-4 receives more limited information and text-rich content is less diverse.

Abstract

from arXiv · show

Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, each containing question-answer pairs for text-rich images. By combining our collected data with previous multi-modal instruction-following data, our model, LLaVAR, substantially improves the LLaVA model's capability on text-based VQA datasets (up to 20% accuracy improvement) while achieving an accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following evaluation also demonstrates the improvement of our model on both natural images and text-rich images. Through qualitative analysis, LLaVAR shows promising interaction (e.g., reasoning, writing, and elaboration) skills with humans based on the latest real-world online content that combines text and images. We make our code/data/models publicly available at https://llavar.github.io/.

1 Introduction

Visual instruction-tuned models struggle with textual details in images, despite textual understanding being integral to everyday visual perception. LLaVAR addresses this gap by adding OCR-based noisy data and GPT-4-generated high-quality conversations to visual instruction tuning.

Motivation: Visual instruction-tuned models struggle with text in images, partly because existing training data is dominated by natural images.The paper argues that textual understanding remains integral to visual perception in everyday life.
Data and approach: 422K noisy instruction-following examples are collected from text-rich images using OCR results and manually written instructions.The noisy data is intended to improve alignment between visual features and the language decoder.
Data and approach: 16K high-quality conversations are generated by prompting text-only GPT-4 with OCR results and image captions.Each conversation can contain multiple turns of question-and-answer pairs for text-rich images.
Model and evaluation: LLaVAR augments LLaVA’s pretraining and fine-tuning data with the collected noisy and high-quality examples.The study also evaluates higher input resolution to better encode small textual details.
Results: LLaVAR significantly enhances text understanding within images while slightly improving performance on natural images.The paper evaluates text-based VQA, GPT-4-based instruction following, and qualitative interactions on online content combining text and images.

2 Related Work

Prior work established instruction tuning for multimodal interaction, but many visual instruction-tuned models remained weak on OCR-dependent tasks. LLaVAR builds on this literature by targeting text-rich image understanding within an end-to-end visual instruction-tuning framework.

Instruction Tuning: Instruction tuning formulates varied tasks as natural-language instructions to improve generalization and support interaction with users.The related work describes human feedback, multitask instruction following, and LLM-generated instruction data.
Terminology: Text-rich images are images containing text, such as posters and book covers, whereas natural images are described as images without text.
Instruction Tuning: GPT-4-based instruction-following evaluation is useful but raises concerns about response-length bias and sensitivity to example order.The paper follows prior work in using this evaluation approach.
Multimodal Instruction Tuning: Existing multimodal instruction-tuning systems use diverse data and training strategies, yet many struggle with tasks requiring OCR capability.Examples include caption-based curation, large-scale image-text pretraining, task transformation, and masked-text prediction.

3 Data Collection

LLaVAR’s data pipeline filters LAION images for text-rich content, produces large-scale OCR-aligned examples, and uses selected coherent images with GPT-4 to create conversational instruction data.

Image Selection: A DiT-base classifier fine-tuned on RVL-CDIP is used to select LAION images likely to contain text while filtering watermark-heavy images.The collection begins by combining natural-image and document data into a binary classification dataset.
Image Selection: CLIP clustering identifies 14 diverse text-rich image clusters, including posters, covers, advertisements, infographics, educational materials, and logos.The clusters are selected after inspecting 100 clusters formed from 50K sampled images.
Noisy Instruction-following Data: 422K deduplicated images from the 14 preferred clusters are used for noisy instruction-following data, with at most 52K examples retained per cluster.The images are processed with PaddleOCR.
Noisy Instruction-following Data: OCR is performed after downsampling images to balance recognition of visible fonts with OCR accuracy, then recognized words are merged into paragraphs geometrically.This avoids extracting tiny fonts that visual encoders may not visibly represent.
GPT-4-based Instruction-following Data: 16K non-overlapping images from four selected clusters are used for GPT-4 prompting because their OCR contains enough visible, coherent sentences.The filtering increases the proportion of book covers and quote images.
GPT-4-based Instruction-following Data: GPT-4 receives two complementary OCR outputs and a BLIP-2 image caption to generate conversational questions and answers.The questions become input instructions and the answers become output responses.

4 Model Architecture and Training

LLaVAR largely follows LLaVA’s architecture, mapping CLIP visual features into the language decoder’s embedding space. Its two-stage training aligns visual features with noisy data first, then trains the instruction-following model with augmented data.

Architecture: CLIP-ViT-L/14 encodes 224^2 images and CLIP-ViT-L/14-336 encodes 336^2 images in the main configurations.Grid features before the final transformer layer are used as visual representations.
Architecture: A trainable projection matrix W transforms visual grid features into the language decoder’s word-embedding space.The language decoder is Vicuna-13B except in the stated ablation study.
High-resolution Extension: A high-resolution visual encoder can produce thousands of patch features, so cross-attention modules let the decoder attend to transformed high-resolution features.This addresses context-length limits when high-resolution features and instruction tokens are combined.
Training: During pretraining, only W is trained for feature alignment while the decoder D remains frozen, allowing noisy data to be used.The pretraining mixture combines LLaVA’s 595K examples with 422K noisy instruction-following examples.
Training: Both training stages generate response tokens from image and instruction tokens, with the second stage training the projection and language-decoder components.The figure caption identifies image tokens as <img>, instruction tokens as <ins>, and response tokens as <res>.

5 Experiments

Experiments show that LLaVAR improves text understanding across text-based VQA tasks, with stronger gains at higher resolution, while ablations clarify the roles of OCR-based data, GPT-4-generated supervision, and visual encoders. Additional evaluations show improved reading of text-rich images, preserved or improved natural-image instruction following, and recognizable-font-size thresholds, alongside limited data diversity.

Text-based VQA: LLaVAR substantially improves the LLaVA baseline on all four text-based VQA datasets and performs best on three of them.The comparison covers ST-VQA, OCR-VQA, TextVQA, and DocVQA; the authors caution that comparisons with other models are not fully fair.
Ablation Study on pretraining/finetuning data: OCR-based pretraining and fine-tuning data each help, while GPT-4-generated data complements them and noisy fine-tuning can impair instruction following.Noisy fine-tuning causes the model to output recognized text while ignoring questions; captions are less advantageous than OCR during pretraining.
Ablation Study on encoders/image resolution: Adding a high-resolution Pix2Struct encoder with doubled high-resolution pretraining examples achieves 15.3% zero-shot accuracy on DocVQA.Other datasets show relatively limited gains, and high-resolution OCR is not necessarily better when images are resized identically.
GPT-4-based instruction-following evaluation: LLaVAR achieves a +3.8 relative improvement on the “Read” dimension for instruction-following questions based on 50 text-rich LAION images.The evaluation uses OCR results and human-annotated captions to construct questions, with GPT-4 scoring responses relative to text-only GPT-4 responses.
Case Study: Recognizable Font Size: LLaVAR recognizes text more reliably than LLaVA across answer heights from 3 to 19 pixels, but accuracy sharply decreases below 7 pixels.The 2242-based model performs better at 3-pixel text, whereas the 3362-based model performs better above 7 pixels.
Transferred Instruction-following Capability: The collected instruction-following data is less diverse and substantial than LLaVA’s, although LLaVAR develops reasoning, writing, and elaboration skills for screenshot-based interaction.The authors attribute the data limitation to fewer captions and the less diverse content of text-rich images.

6 Conclusion

LLaVAR enhances visual instruction-tuned models’ ability to read text in images by augmenting LLaVA with OCR-based and GPT-4-generated instruction-following data.

LLaVAR uses 422K noisy OCR-based examples and 16K high-quality text-only GPT-4 instruction-following examples to augment LLaVA training.The two datasets are used for pretraining and finetuning, respectively.
The model demonstrates superior performance in understanding text within images.

Few-shot Examples

The supplied examples and evaluation passages cover text-rich image understanding, dataset benchmarks, metric choices, resolution effects, and instruction-following behavior. Across these results, LLaVAR data improves performance over LLaVA, especially at higher resolution, while preserving or improving natural-image interaction.

Evaluation results: LLaVAR data significantly improves results on CT80, POIE, and ChartQA, usually more at higher resolution.These extra datasets evaluate OCR, information extraction, and chart question answering.
Ablation studies: Caption-only training improves text recognition, but OCR-based training performs better in the reported experiments.The comparison is between ablation variants using captions alone and OCR-based data.
Evaluation results: 91.42% average accuracy on ScienceQA exceeds LLaVA’s 90.92%, with the largest gain occurring on natural science questions at +1.43%.ScienceQA includes multimodal questions whose images may contain text descriptions and tables.
Resolution analysis: Higher-resolution models benefit more from collected data, suggesting the original LLaVA architecture underutilized that data.
Ablation studies: Combining pretraining and finetuning data improves the trade-off between conversation capability and complex reasoning.Pretraining data improves conversation capability, whereas finetuning-only data hurts conversation capability but increases complex reasoning.