Document Collection Visual Question Answering

1 Introduction

• Prior document-collection research was limited largely to lexical word spotting, without semantics or higher-level information extraction.Source: §1 · p. 1
• Earlier text-aware VQA considered text in natural scenes, and document VQA considered individual documents, but neither addressed image collections.Source: §1 · p. 1
• DocCVQA makes collection-level question answering a retrieval-answering task: systems return answers and IDs of documents containing the information needed to infer them.The authors also introduce an evaluation metric and two baselines for the task.Source: §1 · p. 2

2.1 Document understanding

• Document understanding moved from template-specific rules to learning-based field extraction and deep models combining language, visual features, and graph representations.Source: §2.1 · p. 3
• DocCVQA takes a top-down approach in which a question drives the search, unlike bottom-up extraction of key-value pairs from document features.Source: §2.1 · p. 3

2.2 Document retrieval

• Word spotting searches for query-word instances, whereas the proposed task retrieves documents semantically from the question rather than isolated words.Source: §2.2 · p. 3

2.3 Visual Question Answering

• DocVQA brought VQA to document images; DocCVQA extends that setting to collections, unlike ISVQA's small sets of shared-context real-scene images without text.Source: §2.3 · p. 4

3.1 Data Collection

• The dataset contains 14,362 candidate-registration document images from the Public Disclosure Commission, all using one form template with slight design differences.The forms combine complex layout with handwritten and typewritten text.Source: Images · p. 5
• The authors pose 20 natural-language questions over the collection, using date, text-field, and checkbox constraints to avoid reducing the task to simple word spotting.Examples of constraints include date intervals, inclusion or exclusion of specified values, and checked or unchecked boxes.Source: Questions and Answers · p. 5

3.2 Statistics and Analysis

• Test questions include form fields or constraints absent from the sample set, and the number of relevant documents varies across questions.Figure 2 compares relevant-document counts for sample-set questions in blue and test-set questions in red.Source: Questions and Answers · p. 6, Figure 2 · p. 7

3.3 Evaluation metrics

• The task evaluates evidence retrieval separately from answer performance to check whether answers are grounded in documents containing the needed information.Source: §3.3 · p. 7
• Evaluation uses MAP for positive evidence documents and ANLSL for answers, with ANLSL matching unordered answer lists and penalizing list-length mismatches.ANLSL uses Hungarian matching based on normalized Levenshtein similarity, then divides the matched scores by the longer of the ground-truth and predicted lists.Source: §3.3 · p. 8

4 Baselines

• Both baselines rank documents by question relevance, then answer from the highest-confidence documents, but use distinct text-based and database-based strategies.Text spotting + QA uses word spotting and NLP question answering; Database maps extracted information into key-value relations.Source: §4 · p. 8

4.1 Text spotting + QA

• The Text spotting + QA baseline combines generic methods to provide a starting performance level for assessing future methods.Its stated objective is to measure improvement over this baseline.Source: §4.1 · p. 9
• For retrieval, it keeps nouns and digits from each question, averages their minimum Normalized Levenshtein Distances to OCR words, and uses that score to rank documents.Removing only stopwords can retain words such as “run” that do not occur in the document and can be counterproductive.Source: Evidence retrieval · p. 9
• For answering, BERT predicts an extractive span from OCR tokens serialized in top-left-to-bottom-right order, scoring candidate spans with start and end vectors.The highest-scoring span with end position j ≥ i is selected.Source: Answering · p. 9

4.2 Database approach

• The Database baseline uses commercial OCR to extract and normalize key-value records, marks a document relevant only when all query constraints match, then extracts the requested value.It is presented as a heuristic-based performance target because human evaluation would require checking more than 14k documents per question.Source: §4.2 · p. 10
• The Database approach is rigid, does not allow data modification, is described as not generalizable, and requires manual natural-language-to-SQL query preprocessing.Source: §4.2 · p. 10

5.1 Evidences

• Database retrieval has similar average performance to text spotting, but reaches MAP 0 on question 11 and struggles with complex constraints.The question 11 failure involves a treasurer-name field relation; other gaps involve date ranges, negated values, and multiple acceptable values.Source: §5.1 · pp. 10–11

5.2 Answers

• After six fine-tuning epochs on around 80 DocCVQA samples, BERT answering improves without harming its previous knowledge.The model was also fine-tuned on DocVQA before the DocCVQA sample set.Source: §5.2 · pp. 11–12
• The database approach performs best among the evaluated answering baselines.The authors attribute this to its direct use of key-value relations, while BERT can lose the semantic connection between a field name and value.Source: Table 4 · p. 12, §5.2 · p. 12
• BERT performs poorly on candidate-name questions but better on dates and legislative counties, while database answering depends on correct text and key-value recognition.Source: §5.2 · p. 12
• Neither method answers checkbox question 13, even with correct retrieval.BERT lacks a collection-level perspective and needs contextualized answer text; the database method's logic does not handle this negative checkbox case.Source: §5.2 · p. 13

• DocCVQA introduces a challenging task for VQA and document analysis centered on understanding whole document collections.Source: §6 · p. 13
• Text spotting with extractive QA is less performant but more generic, whereas the key-value baseline performs better yet cannot generalize to other document types.Source: §6 · p. 13
• The proposed next step is a single-stage method that reasons over the collection and provides both answers and positive evidence documents.Source: §6 · p. 13