Source-linked AI summary
SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images
Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, Kuniko Saito
TL;DR
Existing document VQA datasets mostly focus on relationships within single images, leaving multi-image reasoning insufficiently covered. The paper introduces SlideVQA and the unified M3D model, which outperforms existing QA models but remains far behind humans.
Problem
Most existing document VQA datasets focus on single images rather than reasoning across multiple images, despite the need to understand textual, layout, and visual information.
Method
The paper introduces SlideVQA and M3D, an end-to-end model that jointly performs evidence selection and question answering while generating arithmetic expressions.
Results
M3D outperformed existing state-of-the-art QA models on SlideVQA, but remained far below human performance.
Takeaways & Limitations
SlideVQA provides a benchmark for document VQA involving multiple images, multi-hop reasoning, and numerical reasoning.
Takeaways & Limitations
Edited multi-hop questions may differ from questions humans would actually ask, and cross-attention over all evidence candidates can become computationally costly with many input images.
Abstract
from arXiv · showhide
Visual question answering on document images that contain textual, visual, and layout information, called document VQA, has received much attention recently. Although many datasets have been proposed for developing document VQA systems, most of the existing datasets focus on understanding the content relationships within a single image and not across multiple images. In this study, we propose a new multi-image document VQA dataset, SlideVQA, containing 2.6k+ slide decks composed of 52k+ slide images and 14.5k questions about a slide deck. SlideVQA requires complex reasoning, including single-hop, multi-hop, and numerical reasoning, and also provides annotated arithmetic expressions of numerical answers for enhancing the ability of numerical reasoning. Moreover, we developed a new end-to-end document VQA model that treats evidence selection and question answering in a unified sequence-to-sequence format. Experiments on SlideVQA show that our model outperformed existing state-of-the-art QA models, but that it still has a large gap behind human performance. We believe that our dataset will facilitate research on document VQA.
Introduction
SlideVQA addresses the difficulty of answering questions across multiple document images by introducing a dataset and unified model for slide-deck understanding. The dataset requires single-hop, multi-hop, and numerical reasoning, while M3D jointly selects evidence and answers questions, outperforming existing QA models but remaining far below human performance.
- SlideVQA introduces multi-image document VQA in which systems select evidence images from a slide deck and answer a corresponding question.
- SlideVQA requires single-hop, multi-hop, and numerical reasoning over slide images.The examples include arithmetic expressions for deriving numerical answers.
- 2.6k+ slide decks and 14.5k questions make SlideVQA a large benchmark with bounding boxes and arithmetic-expression annotations.Each deck consists of 20 slides, and the annotations support layout understanding and numerical reasoning.
- M3D jointly performs evidence selection and question answering in a unified sequence-to-sequence model while generating arithmetic expressions for numerical reasoning.
- M3D outperformed existing state-of-the-art QA models on SlideVQA, but its performance remained substantially below human performance.
Related Work
Prior document VQA datasets generally assume one relevant image, whereas SlideVQA targets question answering over multiple slide images with multi-hop and numerical reasoning. Related systems combine text, layout, and visual features or use retrieval pipelines, while this work jointly models evidence selection and answering.
- Existing document VQA datasets generally assume that one relevant image contains all facts needed to answer a question.
- SlideVQA differs from DocCVQA through 14.5k versus 20 questions and multi-hop reasoning across slides instead of single-hop reasoning on individual images.
- Single-image document VQA models combine textual, layout, and visual features, whereas this work focuses on reasoning across multiple images.
- Unlike retrieval-and-reading pipelines for DocCVQA, SlideVQA uses multi-task learning to jointly perform evidence selection and question answering.
- No prior dataset supported both multi-hop and numerical reasoning on document images.
The SlideVQA Task and Dataset
SlideVQA formulates document VQA over 20-slide decks, requiring systems to select evidence slides and answer questions involving textual, visual, layout, and numerical information. The dataset combines multi-hop and numerical reasoning with dense annotations of slide content and answer structure.
- Task formulation: SlideVQA asks systems to answer questions about ordered 20-slide decks while selecting the relevant evidence slides.The task separates evidence selection from question answering, but evaluates them for the same slide-deck question.
- Slide annotation: SlideVQA annotates textual and visual regions with bounding boxes assigned to nine categories, including titles, text, diagrams, tables, images, and figures.The collection process selected English-understandable decks containing graphs, tables, figures, or numerical data and reports densely annotated images.
- Question creation: Workers created 12,466 single-hop QA pairs from individual slides and 2,018 multi-hop QA pairs by replacing bridge entities with evidence-based descriptions.Questions were encouraged to use arithmetic operations, counting, and comparisons while avoiding page-number or external-knowledge requirements.
- Dataset scale: 14,484 questions come from 2,619 slide decks containing 52,480 slide images and 890,945 annotated bounding boxes.The split contains 10,617 training questions, 1,652 development questions, and 2,215 test questions, with decks kept within one split.
- Reasoning and answers: 49.3% of questions require multi-hop or numerical reasoning, while 32.4% of answers are multi-span or non-span.The dataset also annotates arithmetic expressions, and 25.5% of numerical questions require arithmetic operations.
- Question distribution: Questions often begin with evidence-search phrases such as “In” and “Regarding,” reflecting the need to search across slide decks.This wording pattern is associated with multi-text document QA.
Our Model
M3D extends a multimodal encoder-decoder to solve SlideVQA by jointly generating answers or arithmetic expressions and selecting evidence pages. It represents slide text, layout, regions, and visual information, then uses generated expressions and a calculator for numerical answers.
- Model overview: M3D unifies evidence selection and question answering in a sequence-to-sequence model trained with multitask learning.It extends FiD initialized with pretrained T5 for the end-to-end SlideVQA task.
- Input representation: Each slide input concatenates a task prefix, question, page number, and region-aware OCR context for the encoder.Region labels are inserted before OCR tokens to identify the role of each semantic region.
- Input representation: The encoder combines token, segment, layout, and visual embeddings to represent slide content and its spatial structure.Layout features encode normalized bounding-box coordinates, while visual features represent region appearance and OCR boxes.
- Answer generation: The answer decoder generates either an answer or an arithmetic expression, choosing between them with an Answer: or Expression: indicator token.When an expression is generated, a calculator computes the final numerical result.
- Evidence selection: The evidence selector shares the decoder architecture and generates selected slide page numbers in a unified output sequence.The model is trained with separate decoder and selector negative-log-likelihood losses.
Experiments
Experiments evaluate SlideVQA through evidence selection, question answering, joint performance, ablations, and human comparisons. M3D generally outperformed baselines, while numerical reasoning and multi-image evidence selection remained challenging.
- Evidence selection: 96.0% Recall@3 was achieved by H-LayoutLMv2 on test-set evidence selection.H-LayoutLMv2 encodes all slides simultaneously with an additional Transformer layer.
- Main-task performance: M3D outperformed pipeline methods on the main task's joint EM/F1 and surpassed them across all question-answering metrics.Joint metrics assess consistency between predicted evidence and answers; the end-to-end model better ignored irrelevant slides than pipeline answer generators.
- Human comparison and dataset demands: All models performed significantly worse than humans, especially on multi-hop reasoning across images and non-span arithmetic answers.The experiments also indicate that text, layout, and visual modalities must be understood jointly, while question-only systems performed worst.
- Ablation study: Performance consistently decreased when individual M3D modules were removed in the ablation study.The result supports the contribution of each evaluated module to M3D performance.
- Numerical reasoning: +10.4% F1 was obtained for arithmetic-type questions when predicting arithmetic expressions instead of numerical values.The authors report that arithmetic-expression generation improved QA and joint performance.
- Evidence-selection methods: M3D's separately generated evidence selection and question answering achieved the highest performance among the evidence-selection methods.The authors report that MultiGen was easier to train than ChainGen because the two tasks were trained separately.
Discussion and Limitations
SlideVQA supports scalable evaluation of multi-image reasoning but has limits in question realism and computational scalability. Its edited multi-hop questions may differ from questions users would naturally ask, and cross-attention over many images can be costly.
- Question realism: Edited multi-hop questions may differ from the questions humans would actually ask the system.The editing method guarantees multi-hop questions and allows the dataset to be extended easily.
- Computational scalability: Cross-attention over all evidence candidates may create computational problems when the number of input images is large.The authors suggest a two-stage selector followed by end-to-end evidence selection and answer generation as a possible remedy.
Conclusion
The paper introduces SlideVQA for understanding multi-image slide decks and M3D for unified evidence selection and question answering. Results show promise, but a large human-performance gap and unresolved multi-image and arithmetic challenges remain.
- Conclusion: SlideVQA targets document VQA over slide decks composed of multiple images, while M3D jointly performs evidence selection and question answering.M3D also enhances numerical reasoning by generating arithmetic expressions.
- Conclusion: A huge gap compared with human performance remains, alongside challenges in multi-hop reasoning across images and generating arithmetic answers.The authors state that the dataset may contribute to developing intelligent assistant agents that comprehend diverse real-world documents.