Source-linked AI summary
V-Doc : Visual questions answers with Documents
Yihao Ding, Zhe Huang, Runlin Wang, Yanhang Zhang, Xianru Chen, Yuzhong Ma, Hyunsuk Chung, Soyeon Caren Han
TL;DR
Existing Document-VQA resources often focus on single-page documents or charts and figures, and the paper identifies no platform for comparing novel models with classical baselines. V-Doc provides a dataset-and-model platform with a question-generation engine and web application; on PubVQA, MAC's reported test performance exceeded LSTM+CNN's.
Problem
Existing Document-VQA datasets often focus on single pages or charts and figures, and no platform represents novel-model performance against classical baselines.
Source: §1 · pp. 1–2Method
V-Doc combines dataset and model storage with a web application, a question-generation engine for PubVQA, and support for uploaded abstractive and extractive models.
Source: §1 · p. 2Results
On the PubVQA test dataset, MAC reported performance of 55.44, compared with 48.78 for LSTM+CNN.
Source: §6.2 · p. 7Takeaways & Limitations
V-Doc provides researchers a platform to represent VQA model results and introduces PubVQA, generated from real-world medical journals.
Source: §7 · p. 7
Abstract
from arXiv · showhide
We propose V-Doc, a question-answering tool using document images and PDF, mainly for researchers and general non-deep learning experts looking to generate, process, and understand the document visual question answering tasks. The V-Doc supports generating and using both extractive and abstractive question-answer pairs using documents images. The extractive QA selects a subset of tokens or phrases from the document contents to predict the answers, while the abstractive QA recognises the language in the content and generates the answer based on the trained model. Both aspects are crucial to understanding the documents, especially in an image format. We include a detailed scenario of question generation for the abstractive QA task. V-Doc supports a wide range of datasets and models, and is highly extensible through a declarative, framework-agnostic platform.
1. Introduction
Document-VQA extracts information from documents to answer natural-language questions, but existing datasets often focus on single pages or charts and figures, and no platform compares novel models with classical baselines. V-Doc combines dataset and model storage with a graphical interface, including a question-generation engine and support for uploaded models.
1. Introduction
- Document-VQA requires detecting document objects and understanding their meanings and relations to predict answers; extractive QA selects document tokens or phrases.Abstractive QA generates answers based on recognized document content and a trained model.Source: §1 · p. 1
- Existing Document-VQA datasets often cover single-page documents, while abstractive QA often focuses on charts or figures rather than whole document pages.The paper also identifies the absence of a platform for comparing novel models with classical baselines.Source: §1 · pp. 1–2
- V-Doc's Dataset Storage includes a question-generation engine that creates PubVQA from multi-page documents collected from the PubMed Open Source Subset.It also provides pre-processed datasets such as FUNSD and allows users to add datasets for training and testing.Source: §1 · p. 2
- The platform lets users upload and test trained abstractive or extractive models on different datasets, then compare them with baselines.The web application uses stored trained models to demonstrate their availability and reliability to target users.Source: §1 · p. 2
- V-Doc combines Dataset Storage, Model Storage, and a Graph User Interface as a platform for Document-VQA.The platform lets users compare uploaded abstractive or extractive models with baselines.Source: §1 · p. 2
2. System Architecture
V-Doc's architecture separates datasets, trained models, and a web application. Its dataset module supports pre-processed data and question generation, while the application passes user inputs to selected models for prediction.
2. System Architecture
- Dataset Storage provides pre-processed datasets and a question-generation engine that creates QA pairs from PubMed documents.The engine splits raw PDFs into images, uses Mask RCNN and Google Vision API to generate scene graphs, then processes them into PubVQA.Source: §2 · p. 2
- Model Storage provides trained Document-VQA models, including MAC for abstractive reading comprehension and LayoutLMv2 for extractive information retrieval.Users can upload models built with commonly used machine-learning frameworks.Source: §2 · p. 2
- The web application accepts an image and question, calls a selected model's prediction function, and displays its answer.The React front end passes inputs to a Flask back end; each model runs on a separate server to avoid dependency conflicts.Source: §2 · p. 2
3. Dataset Storage
Dataset Storage combines preprocessed Document-VQA datasets with an engine that generates PubVQA question-answer pairs from PubMed document images. It builds scene graphs from detected document segments and uses compositional function sequences to generate questions.
3.1.1 Data Collection
- PubMed PDFs are split into images, then a PubLaynet-trained Mask-RCNN detects document segments and assigns bounding boxes and categories.The authors describe the detector’s outputs as reasonable and acceptable because PubLaynet shares the dataset’s domain.Source: §3.1.1 · pp. 2–3
3.1.2 Scene Graph Generation
- Scene-graph preprocessing extracts segment text and assigns reading order, while gap distances support category refinement and relationships among segments.Reading order begins with top-left y-coordinate sorting and swaps neighboring segments when a left-side neighbor appears later in the list.Source: §3.1.2 · p. 3
- The scene graphs also encode relative positions, such as left/right and left-bottom/right-top relations between segments.These attributes represent relations between detected document segments for use by the question-generation engine.Source: §3.1.2 · pp. 3–4
3.1.3 Question generation
- The question-generation engine uses predefined templates and functional programs for position, counting, and existence questions, with expected answer types.For a position question, it filters segments by category, selects a target table or figure, then extracts its caption’s relative location.Source: §3.1.3 · p. 4
3.2. Existing Dataset
- FUNSD-QA contains 199 PDF images with layout categories, bounding boxes, text, and entity links used to form extractive QA pairs.Its predefined Question and Answer categories and links support span-classification QA.Source: §3.2 · p. 4
- PubVQA adds a new abstractive Document-VQA dataset generated from PubMed papers, alongside the existing FUNSD-QA dataset.FUNSD-QA is derived from form-understanding data for span-classification extractive QA.Source: §3.2 · p. 4
4. Model Storage
Model Storage hosts existing and user-uploaded trained models built with commonly used machine-learning frameworks, spanning extractive and abstractive QA. Its provided models include BERT and LayoutLMv2 for extractive QA, and LSTM+CNN and MAC for abstractive QA.
4. Model Storage
- Model Storage manages existing and user-uploaded trained models built on commonly used machine-learning frameworks, across extractive and abstractive QA.Its models serve as baselines for comparison with user-uploaded models.Source: §4 · p. 4
4.1. Extractive QA Models
- For extractive QA, BERT and LayoutLMv2 predict answer start and end indices; the provided models are trained on FUNSD-QA and FUNSD, respectively.BERT uses top linear layers to predict start and end logits, while LayoutLMv2 uses a linear layer over hidden-state outputs.Source: §4.1 · pp. 4–5
4.2. Abstractive QA Models
- The abstractive baselines include LSTM+CNN, which combines question and document-image features, and MAC, whose cells perform multi-stage reasoning.LSTM encodes question text and a pretrained CNN extracts visual features; MAC cells use control, read, and write units.Source: §4.2 · p. 5
5. Web Application
V-Doc’s web application lets users choose models and document images, submit questions, and view predicted answers. Its workflow also supports uploading models and datasets and routes requests through a backend.
5.1. Graphical User Interface
- The GUI presents model selection, image selection, question input, and answer representation, with predefined templates and custom questions supported.The returned answer is displayed with the file name and question in a results table.Source: §5.1 · pp. 5–6
5.2. Backend Dependencies
- The backend processes uploaded images and questions for prediction, using dependencies for image handling, data preparation, model preprocessing, and request-response handling.The listed stack includes PyTorch and TensorFlow, Transformers, and Flask.Source: §5.2 · p. 6
5.3. User Instruction
- Users may upload models and datasets, select existing stored models and datasets, then enter questions to receive predictions.Uploading a model requires checking that its dataset is already in Dataset Storage.Source: §5.3 · p. 6
6. Evaluation
The evaluation reports extractive results on FUNSD-QA and abstractive results on PubVQA. The multimodal extractive baseline and MAC abstractive model outperform their respective alternatives in the reported comparisons.
6.1. Extractive QA performance
- 11 BLUE for LayoutLMv2 versus 9.37 BLUE for BERT on FUNSD-QA; the paper reports improved extractive QA performance with multimodal features.The models predict answer start and end indices; the LayoutLMv2 model encodes text, bounding boxes, and visual features.Source: Table 3 · p. 6, §6.1 · p. 6
6.2. Abstractive QA performance
- 55.44 on PubVQA test versus 48.78 for LSTM+CNN; MAC achieves the better reported abstractive QA performance.The tested abstractive baselines are MAC and LSTM+CNN.Source: Table 4 · p. 6, §6.2 · p. 7
7. Conclusion
The conclusion presents V-Doc as a web platform for applying stored extractive and abstractive VQA models and introduces PubVQA, generated from real-world medical journals.
7. Conclusion
- V-Doc provides a web application for predicting answers with uploaded trained models, while its model storage includes extractive and abstractive baselines.Users can also upload other trained models to demonstrate their performance.Source: §7 · p. 7
- PubVQA is a new dataset generated from real-world medical journals using an extendable question-generation engine.Source: §7 · p. 7
- The authors plan to add trained baseline models and question-generation templates to improve dataset quality.Source: §7 · p. 7