Source-linked AI summary
Vision And Text Transformer For Predicting Answerability On Visual Question Answering
Tung Le, Huy Tien Nguyen, Le Minh Nguyen
TL;DR
Answerability in VQA is introduced as a regression problem because binary mappings from VQA predictions do not reflect the task’s essence. VT-Transformer combines BERT-based text features and Vision Transformer image features to predict answerability, outperforming competitive baselines on VizWiz 2020. The formulation uses binary-valued labels for unanswerable and answerable samples.
Problem
Answerability is a multimodal VQA task, but prior approaches derive binary scores by mapping VQA predictions to an “unanswerable” label.
Method
VT-Transformer combines pre-trained BERT and Vision Transformer features and predicts answerability with a regression model.
Results
VT-Transformer outperforms competitive baselines on the VizWiz 2020 Answerability task.
Takeaways & Limitations
Regression-based Vision-Text Transformer modeling provides an effective and robust approach for Answerability on Visual Question Answering.
Takeaways & Limitations
The model defines answerability labels as s_i = {0, 1}, corresponding to unanswerable and answerable samples.
Abstract
from arXiv · showhide
Answerability on Visual Question Answering is a novel and attractive task to predict answerable scores between images and questions in multi-modal data. Existing works often utilize a binary mapping from visual question answering systems into Answerability. It does not reflect the essence of this problem. Together with our consideration of Answerability in a regression task, we propose VT-Transformer, which exploits visual and textual features through Transformer architecture. Experimental results on VizWiz 2020 dataset show the effectiveness and robustness of VT-Transformer for Answerability on Visual Question Answering when comparing with competitive baselines.
1 Introduction
Answerability on Visual Question Answering is framed as a distinct multimodal task for estimating how answerable image-question samples are. The paper replaces binary mappings derived from VQA systems with regression and proposes VT-Transformer to model visual-textual relationships.
- Motivation: Answerability estimates whether image-question samples can be answered, helping filter unanswerable questions before running a complete VQA model.The paper presents this as a way to save computational cost in question answering.
- Problem formulation: Traditional approaches append an “unanswerable” label to the VQA vocabulary and map predictions to binary answerability scores.This formulation treats Answerability as classification rather than directly modeling graded answerability.
- Problem formulation: Unanswerable samples include poor-quality images, ambiguous questions, and inconsistent annotator data, requiring visual-textual features and their relationships to be modeled.These examples motivate treating Answerability as a problem distinct from predicting a suitable VQA answer.
- Proposed direction: The paper proposes VT-Transformer, a regression model combining Transformer-based text and vision representations with pre-trained models.The stated contributions include introducing the Answerability problem, proposing a Vision-Text Transformer, and evaluating it on VizWiz 2020.
2 Methodology
VT-Transformer extracts textual and visual features with BERT and Vision Transformer components, projects them into a shared space, and combines them to regress an answerability score. Its regression objective uses mean squared error rather than VQA-style cross-entropy.
- 2.1 Question embedding: BERT embeds each question, using the [CLS] representation as the question feature.Questions are marked with [CLS] and [SEP] tokens before embedding.
- 2.2 Image embedding: Vision Transformer splits each image into patches, adds positional information, and uses a fully connected layer to extract visual features.The image is treated as a sequence of visual-region tokens.
- 2.3 Vision-Text transformer answerability model: Image and question features are projected into the same dimension space, combined by multiplication or concatenation, and passed through a fully connected layer with a sigmoid to predict answerability.The vector operation is the multimodal fusion step in the VT-Transformer architecture.
- 2.3 Vision-Text transformer answerability model: The model uses mean squared error as its loss because Answerability is formulated as a regression task rather than VQA classification.The paper contrasts this objective with cross-entropy used in VQA.
- 2.3 Vision-Text transformer answerability model: Answerability labels are defined as s_i = {0, 1}, corresponding to unanswerable and answerable samples.This label definition bounds the target values used by the regression formulation.
3 Experiment
Experiments evaluate VT-Transformer and strong baselines on VizWiz 2020 using average precision and F1-score, then assess architectural components and model size.
- 3.1.1 VizWiz dataset: 27% of VizWiz training samples and 32% of validation samples are unanswerable, motivating answerability evaluation on this dataset.VizWiz introduced the Answerability problem and is used for all experiments; the test set is confidential and results are evaluated online through EvalAI.
- 3.1.2 Evaluation metric: Average precision is the recommended VizWiz metric, computed as a weighted mean of precisions under a precision-recall curve.Rn and Pn denote recall and precision at the n-th threshold, respectively.
- 3.2 Results: Pre-trained Vision and Text Transformer models extract features, while fully connected layers normalize and reduce their dimensions.Experimental settings are documented in Table 2, and the baseline comparison includes VWTest5 and BERT-RG-Regression.
- 3.2 Results: VT-Transformer outperforms strong baselines in Average Precision and F1-score on VizWiz 2020.The comparison is reported in Table 3; the authors attribute the improvement in Average Precision to regression formulation and note a 25% F1-score improvement despite annotation dependence.
- 3.3 Ablation Study: Vision Transformer image embeddings outperform ResNet and VGG by approximately 10% on Average Precision and 3% on F1-score.The ablation also finds multiplication better than concatenation for combining visual and textual features.
- 3.3 Ablation Study: The larger B-16 model is better than VT-16, while the smaller VT-16 still performs highly against two baselines.B-16 and VT-16 are approximately 2400MB and 460MB, respectively, and this size comparison excludes pre-trained parameters.
4 Conclusion
The paper concludes that VT-Transformer combines regression-based Answerability with Vision and Text Transformer architectures and pre-trained models. Experiments and ablations show competitive-baseline superiority on VizWiz 2020.
- 4 Conclusion: VT-Transformer integrates Vision and Text Transformer architectures with pre-trained models for regression-based Answerability.The approach is intended to understand multimodal image-question data.
- 4 Conclusion: Experiments and ablation studies show that the model outperforms competitive baselines on the VizWiz 2020 dataset.