PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing the Efficacy of an LLM-Only Approach for Image-based Document Question Answering

Nidhi Hegde, Sujoy Paul, Gagan Madan, Gaurav Aggarwal

arXiv:2309.14389v1cs.CVcs.AI

TL;DR

Document QA systems combine visual and language components, but their relative contributions remain unclear. This paper evaluates language-only QA by serializing image text in reading order for an LLM. The approach achieves near-SOTA results on varied benchmarks, while its reading-order strategy may not transfer equally well to other document types.

  • Problem

    The relative contributions of vision encoders and language models to document question answering remain unclear.

    Source: §1 · p. 2

  • Method

    The paper sends image text serialized in reading order to an LLM, without encoding visual components or other layout information.

    Source: §1 · p. 2

  • Results

    The reading-order-plus-LLM approach achieves near-SOTA performance across benchmark datasets involving text in images.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The analysis can guide model and dataset choices, including identifying tasks for which a vision encoder is essential.

    Source: §6 · p. 8

    A basic reading-order strategy may not work as well for other document types, although suitable heuristics may be feasible for somewhat homogeneous documents.

    Source: §4.5 · p. 6

Abstract

from arXiv · show

Recent document question answering models consist of two key components: the vision encoder, which captures layout and visual elements in images, and a Large Language Model (LLM) that helps contextualize questions to the image and supplements them with external world knowledge to generate accurate answers. However, the relative contributions of the vision encoder and the language model in these tasks remain unclear. This is especially interesting given the effectiveness of instruction-tuned LLMs, which exhibit remarkable adaptability to new tasks. To this end, we explore the following aspects in this work: (1) The efficacy of an LLM-only approach on document question answering tasks (2) strategies for serializing textual information within document images and feeding it directly to an instruction-tuned LLM, thus bypassing the need for an explicit vision encoder (3) thorough quantitative analysis on the feasibility of such an approach. Our comprehensive analysis encompasses six diverse benchmark datasets, utilizing LLMs of varying scales. Our findings reveal that a strategy exclusively reliant on the LLM yields results that are on par with or closely approach state-of-the-art performance across a range of datasets. We posit that this evaluation framework will serve as a guiding resource for selecting appropriate datasets for future research endeavors that emphasize the fundamental importance of layout and image content information.

1. Introduction

The paper examines whether document question answering can be treated as a language-modeling task, since multimodal models leave the contributions of vision and language unclear. It serializes image text by reading order and finds that an LLM-only approach can approach state-of-the-art performance.

1. Introduction

  • Reading-order text fed to pretrained LLMs comes close to SOTA without visual encoding or position information beyond reading order.Figure 1b presents the comparison with SOTA.Source: Figure 1 · p. 1, §1 · p. 2
  • The motivation is that multimodal systems make it difficult to distinguish gains from stronger vision encoders from gains due to language-model reasoning.Some questions may be answerable from language-model knowledge alone, while others may require image content or layout.Source: §1 · p. 2
  • The work investigates whether language models can answer document questions without a vision encoder, and how well they do across task types.Its analyses include ablations and qualitative and quantitative study of when text-only models work or need more visual information.Source: §1 · p. 2
  • The paper frames its goal as evaluating text tokenization with pretrained language models, not proposing a vision-free algorithm for every task or reducing VLM computation.The authors position the analysis as a yardstick for deciding whether tasks need additional image and layout information.Source: §1 · p. 2
  • The approach models DQA as pure language modeling by tokenizing image text into a reading-ordered sequence.The paper studies this approach and quantitatively correlates performance with reading-order perplexity, task lengths, and answer contents.Source: §1 · p. 2

2. Related Work

Prior document-understanding methods are grouped into OCR-based approaches that combine text and visual inputs and OCR-free approaches that rely on images. The paper situates text-only modeling as an exploration of document QA using OCR text in a specific order.

2. Related Work

  • Prior work is divided into OCR-based methods and OCR-free methods.The two groups differ in whether they use explicit OCR text alongside the image or rely on visual information alone.Source: §2 · p. 2
  • OCR-based systems use raw images and OCR text with positions, often fusing text and image tokens through architectures such as UDOP and FormNet.Some methods also incorporate detected objects.Source: §2 · p. 2
  • OCR-free methods rely on image information and often pretrain models to learn text implicitly; Pix2Struct falls short on text-heavy tasks.Examples include DONUT, which uses reading-order prediction, and Pix2Struct, which learns to predict HTML that can render an image.Source: §2 · p. 2
  • OCR-free approaches may require substantially higher image resolution to read text, increasing computational cost.Source: §2 · p. 3
  • This work explores text-only models using OCR-derived textual context in a specific order for document VQA.Source: §2 · p. 3

3. Modeling DocumentQA as an LLM-only Task

The paper converts document QA into language modeling by concatenating OCR text in reading order and supplying it with the question to an LLM. Reading-order quality matters, and the setup shows reasonable zero-shot performance with gains after fine-tuning.

3. Modeling DocumentQA as an LLM-only Task

  • The evaluation intentionally removes image and absolute-position information, focusing on the language model’s performance without them.The setup contrasts with typical document QA frameworks that use image patches and/or OCR tokens with spatial embeddings.Source: §3 · p. 3
  • Reading order converts two-dimensional document text into a coherent one-dimensional sequence, though determining the order can be difficult for complex layouts.The described extraction models generally use layout information rather than content features.Source: §3 · p. 3
  • Reading-order errors may occur in complex documents, and dataset-specific rule-based heuristics can improve ordering and QA performance.Source: §3 · p. 3
  • The method concatenates text obtained in reading order into an OCR context, then prompts an LLM with that context, a question, and an answer placeholder.The prompt is designed to work with encoder-decoder or decoder-only LLMs; the experiments use Flan-T5.Source: §3 · p. 3
  • Text-only performance depends significantly on reading-order extraction, while the scheme has reasonable zero-shot performance and improves after fine-tuning.Fine-tuning aligns predictions with the data and helps the model reason from irregular text caused by ordering errors.Source: §3 · p. 3

4. Experiments

The experiments test reading-order text with Flan-T5 across six benchmark datasets and model sizes, including zero-shot and fine-tuned settings. Fine-tuned LLM-only results approach or exceed vision-based baselines on several tasks, while reading order, context length, and answer availability relate to performance.

4.1. Datasets

  • The method was evaluated on six benchmark datasets spanning documents, infographics, tabular data, illustrative science diagrams, book covers, and scene text; reported metrics include exact match accuracy, ANLS, VQA accuracy, and relaxed accuracy.Source: §4.1 · pp. 3–4

4.2. Models and Implementation Details

  • The models use OCR text serialized in reading order with the question as input; the encoder-decoder produces the answer.The experiments use instruction-tuned Flan-T5 models, with input sequence length set to 1024 for most datasets and 128 for OCR-VQA.Source: Figure 2 · p. 4, §4.2 · p. 4

4.3. Zero-Shot DocumentQA

  • Zero-shot FlanT5-XXL was 4.2% behind fine-tuned Pix2Struct-B on InfoVQA and 13.4% behind UDOP, despite using no vision encoder or dataset fine-tuning.Zero-shot performance increased with model size.Source: §4.3 · p. 4, Table 2 · p. 5

4.4. Finetuning for DocumentQA

  • Fine-tuned FlanT5 models reach 70.5% on OCR-VQA, just 0.8% behind SOTA; the LLM-only approach surpasses LaTR-L by 2.5% on TextVQA, Pix2Struct-L by 5.2% on InfoVQA, and MATCHA by 25.6% on AI2D.Source: §4.4 · p. 5

4.5. Reading Order Quality

  • A raster-scan reading order significantly improved DocVQA performance, while shuffling the order considerably reduced performance across datasets.The authors caution that this basic strategy may not work as well for other document types; reading-order effects were greater with longer contexts.Source: §4.5 · p. 6, Figure 3 · p. 6

4.6. Effect of OCR Context Length

  • Longer image text was associated with more errors overall, especially on InfoVQA, while context-length differences were small for TextVQA and AI2D.The authors note that visual understanding beyond text may matter for TextVQA and AI2D.Source: §4.6 · p. 6

4.7. What type of questions cannot be answered?

  • Incorrect answers had a lower percentage of answer text present in the image than correct answers, after excluding Yes/No questions and specified OCR-VQA genre questions.The method is designed to answer from image text or world knowledge.Source: §4.7 · pp. 6–7, Figure 5 · p. 7

5. Discussion

The discussion introduces Reading Order Perplexity as a heuristic and reports that zero-shot perplexity tracks model performance. It uses question types to distinguish tasks an LLM-only model can address from those needing visual reasoning.

5. Discussion

  • Reading Order Perplexity measures the perplexity of predicted tokens given the reading-order text passed to the model as context.Zero Shot Perplexity is the average reading order perplexity of predictions in a zero-shot setting.Source: §5 · p. 7
  • Correct predictions have significantly lower zero-shot perplexity than incorrect ones, and lower perplexity correlates with better fine-tuned and SOTA performance.The authors suggest zero-shot perplexity can help estimate model potential on unseen tasks.Source: Figure 7 · p. 8, §5 · p. 8
  • The authors distinguish five question types: direct extraction, text plus world knowledge, visual-only, world-knowledge-only, and combined visual-text reasoning.The figure presents example document-image question-answer pairs for different question types.Source: §5 · p. 8, Figure 6 · p. 8
  • LLM-only models address direct extraction, text-plus-world-knowledge, and world-knowledge-only questions; the authors recommend examining visual-dependent question types when assessing vision encoders.They suggest that the strong performance across datasets indicates many questions fall into categories that do not require visual reasoning.Source: §5 · p. 8

6. Conclusion

The conclusion frames image-based document question answering as an LLM-only task by serializing image text in reading order. The authors report near-SOTA performance across varied benchmarks and describe the analysis as guidance for model and dataset choices.

6. Conclusion

  • The paper models document question answering without a vision encoder by sending image text as an ordered token sequence to an LLM.It analyzes factors affecting this approach and reports near-SOTA performance across varied visual question-answering benchmarks involving text in images.Source: §6 · p. 8
  • The authors hope the analysis will guide practitioners' model choices and researchers' dataset choices when developing models for which a vision encoder is essential.Source: §6 · p. 8

A. Multi-Tasking

The multi-tasking experiments fine-tune FlanT5-B and FlanT5-XXL on a mixture of four datasets using two sampling strategies. The reported strategy differences vary by model scale, with some XXL mixture results exceeding the paper's best reported results.

A. Multi-Tasking

  • The experiments fine-tune FlanT5-B and FlanT5-XXL on DocVQA, OCRVQA, InfoVQA, and TextVQA using uniform or normalized sampling.Uniform sampling first selects a dataset with equal probability; normalized sampling pools the datasets before sampling examples.Source: A. Multi-Tasking · p. 11
  • For FlanT5-XXL, normalized sampling consistently performs significantly better, and mixture training further improves on the paper's best reported results for some datasets.The reported examples include OCR-VQA and DocVQA.Source: A. Multi-Tasking · p. 11, Table 4 · p. 11
  • For FlanT5-B, uniform sampling performs slightly better on most datasets, except OCR-VQA, while the two strategies perform closely overall.The authors attribute the exception to normalized sampling drawing more examples from OCR-VQA.Source: A. Multi-Tasking · p. 11

B. Test Set Results

The section reports test-set performance for the best-performing model across datasets, using the respective evaluation servers. It notes that ChartQA and AI2D test scores were already reported in the main paper.

B. Test Set Results

C. Dataset-wise analysis

The datasets span questions answerable from OCR text and world knowledge to those requiring visual content or layout, with each dataset showing a different mix of needs.

  • The analysis distinguishes five question types: direct text extraction, text-plus-world-knowledge reasoning, visual-only answers, world-knowledge-only answers, and reasoning over visual and text content.Source: C. Dataset-wise analysis · p. 11
  • TextVQA questions often combine OCR with visual understanding of relative positions, object structure, symbols, or font sizes.Source: C. Dataset-wise analysis · p. 11
  • ChartQA questions require either OCR extraction or chart understanding, while some rely only on world knowledge beyond the graph.Source: C. Dataset-wise analysis · p. 11
  • OCRVQA performance is on par with or even better than state-of-the-art models using the LLM-only approach.Source: C. Dataset-wise analysis · p. 11
  • AI2D is vision-heavy, but most questions can be answered from OCR context and world knowledge, with fewer requiring visual-plus-text reasoning.Source: C. Dataset-wise analysis · p. 11
  • DocVQA requires both OCR text and layout for accurate answers, yet Raster Scan reading order performs decently without layout information.Source: C. Dataset-wise analysis · p. 11
  • InfoVQA often combines text with world knowledge, although some questions depend critically on visual information.Source: C. Dataset-wise analysis · p. 11
Loading 2309.14389v1…