Source-linked AI summary
BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, Zhuowen Tu
TL;DR
VLMs struggle to interpret text-rich images because fixed query embeddings can limit the visual information passed to language models. BLIVA combines learned query embeddings with projected image patches, improving performance across text-rich and general VQA benchmarks while showing a limitation on numerical symbols.
Problem
VLMs struggle to interpret textual details in images, while fixed query-embedding abstractions limit the information extracted for language models.
Method
BLIVA augments InstructBLIP by combining learned query embeddings with additional encoded patch embeddings projected into the language model.
Results
BLIVA improves text-rich and general VQA performance and achieves the best average performance among current methods on MME perception and cognition tasks.
Takeaways & Limitations
Mixing learned query embeddings with encoded patch embeddings supports BLIVA’s performance on both text-rich and general visual question answering.
Takeaways & Limitations
Human evaluation found that BLIVA struggles to decipher numerical symbols in images, possibly because these symbols receive reduced pixel representation.
Abstract
from arXiv · showhide
Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. Our code and models are freely accessible at https://github.com/mlpc-ucsd/BLIVA.
Introduction
VLMs have advanced open-ended VQA, but models using fixed query embeddings struggle with text inside images. BLIVA addresses this by combining learned query embeddings with encoded patch embeddings and evaluates the design across text-rich and general VQA settings.
- Introduction: VLMs have advanced open-ended visual question answering by adding visual understanding to large language models.
- Introduction: Fixed query-embedding abstraction modules can limit models’ ability to interpret textual details within images.
- Introduction: BLIVA combines learned query embeddings with encoded patch embeddings to provide additional visual information to the language model.
- Introduction: BLIVA improves text-in-image understanding while maintaining robust performance on general VQA benchmarks and achieving the best performance on the MME benchmark among previous methods.
Related Work
Prior multimodal systems connect vision and language through query-based extraction, direct patch-feature alignment, or broader instruction-tuning frameworks. BLIVA’s surrounding work includes these architectural and training paradigms, alongside multimodal datasets and applications.
- Related Work: The paper follows a two-stage multimodal training paradigm while focusing on an end-to-end model for text-rich and general VQA benchmarks.
- Related Work: Multimodal instruction tuning extends instruction-based generalization methods from natural language processing to image-based tasks.
- Related Work: Flamingo and BLIP-2 use query-based visual feature extraction, while LLaVA directly connects encoded image patches to the language model.
- Related Work: The study uses instruction-tuning data following InstructBLIP’s prompts to evaluate the effectiveness of additional encoded patch embeddings.
Method
BLIVA combines query-derived and directly projected patch features before sending them to the language model. Its training uses staged visual-language alignment, and its evaluation includes both standard VQA benchmarks and a YouTube-thumbnail dataset.
- Method: BLIVA sends image features separately through a Q-Former and projection layer, concatenates the resulting embeddings, and feeds them to the language model after the question embedding.
- Method: The model uses a two-stage scheme: image-text pre-training first aligns visual information, followed by instruction tuning for finer visual question answering.
- Method: BLIVA’s patch-feature branch uses a compact 0.5M image-caption pre-training dataset rather than BLIP-2’s 129M-example dataset.
- Method: YTTB-VQA contains 400 YouTube thumbnail question-answer pairs across 11 categories for evaluating visual perception of text-rich images.
- Method: BLIVA provides detailed captions and answers to visual questions in sample YTTB-VQA scenarios.
Experiment
BLIVA is evaluated against alternative single-image-embedding approaches across text-rich VQA, general VQA, MME, and YouTube-thumbnail scenarios. The experiments examine overall efficacy, component influence, and real-world thumbnail question answering.
- Text-rich VQA: BLIVA consistently improves over InstructBLIP across text-rich VQA datasets and achieves the best average performance among previous methods.The OCR-VQA comparison is not zero-shot because InstructBLIP included OCRVQA during training.
- YouTube-thumbnail evaluation: The YTTB-VQA dataset demonstrates BLIVA’s use for detailed image captioning, concise visual summarization, and subsequent detailed visual questions.The scenarios focus on YouTube thumbnails and associated question-answer interactions.
- General VQA: BLIVA significantly outperforms InstructBLIP on 7 of 8 general VQA benchmarks, while nearly matching it on VizWiz.Improvements are reported for VSR, IconQA, TextVQA, Visual Dialog, Hateful Memes, MSRVTT, and Flickr30K.
- MME benchmark: BLIVA achieves the best average performance on both perception and cognition tasks in the MME benchmark.It outperforms InstructBLIP on text-rich tasks including OCR, Poster, Numerical Calculation, Text Translation, and Code.
2. How do the individual components of our method influence its success?
Ablations test patch embeddings, pre-training, and LLM fine-tuning separately within BLIVA. Patch embeddings and two-stage training provide the main benefits, while LoRA-based LLM fine-tuning is less effective.
- Text-rich VQA ablations: Patch embeddings improve text-rich VQA performance across most tasks, supporting their role in providing more detailed visual information.ST-VQA and OCR-VQA are exceptions, while the authors attribute limited extraction by query embeddings to the broader benefit.
- General VQA ablations: Patch embeddings significantly improve all general VQA benchmarks except Hateful Memes and VizWiz.The authors associate these exceptions with tasks that can rely on global-level query-embedding information.
- Pre-training: Pre-training improves VSR, VisDial, Hateful Memes, and MSRVTT substantially, while leaving other general VQA results roughly unchanged.The ablations confirm the necessity of the two-stage training procedure.
- LLM fine-tuning: LoRA-based LLM fine-tuning produces less improvement than the best model and reduces performance on many tasks.The experiment combines LLM LoRA fine-tuning with Q-former and encoded patch embeddings.
3. How does BLIVA enhance the recognition of YouTube thumbnails?
BLIVA performs strongly on YouTube thumbnail VQA, extracting visual information beyond captions alone. Its qualitative results also show OCR and localization abilities on text-rich images.
- YouTube Thumbnail Evaluation: BLIVA achieves the best performance on the YouTube thumbnail dataset.The dataset contains question-answer pairs associated with YouTube thumbnails.
- YouTube Thumbnail Evaluation: BLIVA extracts extra visual information from thumbnails beyond information available from YouTube captions alone.This supports using visual perception rather than relying solely on caption text.
- Qualitative Analysis: BLIVA demonstrates OCR and localization abilities on real-life scenes, movie posters, webpages, and memes.The model identifies text and objects within text-rich images.
Conclusion
BLIVA combines learned query embeddings with encoded image patch embeddings as a visual assistant. The model improves general and text-rich VQA performance, but struggles with numerical symbols in images.
- Conclusion: BLIVA assists learned query embeddings with encoded image patch embeddings as a visual assistant.The paper presents this combination as its central design.
- Conclusion: BLIVA improves performance on both general and text-rich VQA benchmarks.The paper also reports superior performance on academic benchmarks and qualitative real-world examples.
- Conclusion: BLIVA struggles to decipher numerical symbols in images, possibly because these symbols receive reduced pixel representation.The paper identifies this as an area requiring future work.
Official Version
The official AAAI proceedings version will be notified and made accessible through the BLIVA GitHub repository.
- Official Version: The paper states that its official AAAI proceedings version will be notified.
- Official Version: The official version will be accessible through the BLIVA GitHub repository.
- Official Version: The repository named for access is github.com/mlpc-ucsd/BLIVA.
Time and Memory Complexity
BLIVA's reported resource evaluation covers training time, memory, and inference time. The authors state that it outperforms InstructBLIP without sacrificing much computation power.
- Time and Memory Complexity: BLIVA outperforms InstructBLIP without sacrificing much computation power.
- Time and Memory Complexity: Training time is reported for one epoch using 8 NVIDIA A6000 GPUs with batch size 24.
- Time and Memory Complexity: Inference time is measured on an NVIDIA A100, while Table 8 compares time and memory overheads.
Text Fonts in images
BLIVA’s text-capture performance depends on image-text contrast, while its broader design choices and examples support handling both textual and spatial visual information.
- Text Fonts in images: Light-colored fonts, especially white or pale green, outperform darker blue fonts on grayscale backgrounds because color contrast affects performance.The evaluation used Times New Roman and Impact across white, red, pale green, and blue fonts.
- Encoding variants: A linear projection for localized positional details outperforms other BLIVA encoding variants across four representative datasets.The ablation concludes that a simple projector provides better generalization.
- Data and preprocessing: BLIVA’s evaluation follows prior work’s data and preprocessing setup, with most training datasets retained except for unavailable LAION-115M content and some image-captioning samples.The retained data include MSCOCO, TextCaps, VQAv2, OKVQA, A-OKVQA, OCR-VQA, and LLaVA-Instruct-150K.
- Data and preprocessing: Training preprocessing randomly crops and resizes images to 224 * 224, may flip them horizontally, converts them to tensors, and normalizes them.Evaluation preprocessing resizes images with bicubic interpolation, converts them to tensors, and applies the same normalization values.
- Real-world examples: Examples show BLIVA recognizing text in road signs, shopping products, memes, web pages, movie posters, and packaging while also identifying spatial relationships.The examples are presented as evidence of practical interpretation of everyday visual information.
- Real-world examples: The YTTB-VQA dataset distributes questions across 11 categories, including technology, sports, entertainment, food, news, history, music, nature, cars, and education.The category chart describes a broad range of YouTube-thumbnail content.