Source-linked AI summary
Did the Model Understand the Question?
Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, Kedar Dhamdhere
TL;DR
Deep QA models may achieve high accuracy while ignoring important question terms, making test-set accuracy an incomplete measure of their behavior. The paper applies attribution-based analysis to image, table, and passage QA, then uses exposed weaknesses to craft or strengthen adversarial attacks. The strongest reported attacks reduce Visual QA accuracy from 61.1% to 19% and tabular QA accuracy from 33.5% to 3.3%.
Problem
High test-set accuracy is informative only when tests represent the real-world task, yet large datasets are difficult to check and models may rely on unimportant question terms.
Method
The paper uses Integrated Gradients to attribute predictions to question words, identify model-specific weaknesses, and generate targeted adversarial questions.
Results
The analysis finds ignored important question terms across three QA tasks; attacks reduce Visual QA accuracy from 61.1% to 19% and tabular QA accuracy from 33.5% to 3.3%, while attribution strengthens passage attacks.
Takeaways & Limitations
Attributions augment accuracy measurements by surfacing erroneous model logic and can help developers improve model quality and evaluate whether responses should be trusted.
Takeaways & Limitations
The attribution analysis is specific to deep-learning-based systems, and its scores are defined relative to an empty-question baseline while the image, table, or passage context remains fixed.
Abstract
from arXiv · showhide
We analyze state-of-the-art deep learning models for three tasks: question answering on (1) images, (2) tables, and (3) passages of text. Using the notion of \emph{attribution} (word importance), we find that these deep networks often ignore important question terms. Leveraging such behavior, we perturb questions to craft a variety of adversarial examples. Our strongest attacks drop the accuracy of a visual question answering model from $61.1\%$ to $19\%$, and that of a tabular question answering model from $33.5\%$ to $3.3\%$. Additionally, we show how attributions can strengthen attacks proposed by Jia and Liang (2017) on paragraph comprehension models. Our results demonstrate that attributions can augment standard measures of accuracy and empower investigation of model performance. When a model is accurate but for the wrong reasons, attributions can surface erroneous logic in the model that indicates inadequacies in the test data.
1 Introduction
The paper uses attribution-based analysis to expose question-answering models’ reliance on unimportant words across image, table, and passage tasks. These weaknesses support targeted adversarial questions and reveal why accuracy can be misleading.
- Analysis workflow: The study analyzes three question-answering models by attributing predictions to question words, identifying weaknesses, and crafting adversarial questions.The workflow applies Integrated Gradients, visualizes attributions, and exploits revealed weaknesses.
- Visual QA: Visual QA retains more than 50% of its original accuracy after deleting every question word except “color” from validation questions.The model also under-relies on important question words and relies largely on the image.
- QA on tables: 33.5% to 3.3%: content-free phrases sharply reduce Neural Programmer’s tabular question-answering accuracy.Neural Programmer’s operation selections are more influenced by content-free words than by nouns or adjectives.
- Reading comprehension: Reading-comprehension attacks are 50% more likely to succeed when added sentences contain the question’s top-attributed nouns.Attribution-guided perturbations strengthen attacks that add supposedly irrelevant sentences to passages.
- Conclusion: Across all networks, ignored important question terms expose overstability that can be tested by measuring accuracy as question words are systematically dropped.The paper proposes improving training data or introducing an inductive bias, while using attribution visualizations to help assess trustworthiness.
2 Related Work
The paper contrasts model-independent adversarial evaluation with attribution-guided analysis that inspects model logic. Its method targets high-attribution question terms across three QA tasks and also improves existing passage attacks.
- Prior adversarial evaluation: Jia and Liang’s method augments reading-comprehension test sets with crowdsourced passage perturbations intended to fool networks, independently of model specifications.Their approach can improve evaluation without necessarily fixing the model itself.
- Attribution-based analysis: This paper uses attributions to inspect network logic, identify important question terms receiving tiny attribution, and craft targeted question attacks.The approach exploits model-specific strengths and weaknesses rather than remaining model-independent.
- Scope and distinction: The study analyzes one deep-learning QA system for each of three tasks, whereas Jia and Liang focus exclusively on reading comprehension.The attribution technique is specific to deep-learning-based systems.
- Connection to prior work: Attribution-derived insights improve the efficacy of Jia and Liang’s attacks on reading-comprehension systems.The paper directs readers to examples in table 4.
- Attribution methods: Integrated Gradients is chosen for ease and efficiency of implementation and for its axiomatic justification, although other attribution methods could be used.The paper notes that model-agnostic techniques are also possible.
3 Integrated Gradients (IG)
The paper uses Integrated Gradients (IG) to assign question-word attributions relative to an empty-question baseline. It validates these attributions through perturbations: high-attribution terms affect predictions, while low-attribution terms often do not.
- Integrated Gradients (IG): Integrated Gradients assigns each question term an influence on a model’s prediction relative to a baseline input.The question terms are represented as vectors, and the attributions quantify their effects on the prediction probability.
- Integrated Gradients (IG): The paper uses an empty question as the baseline while preserving the instance’s image, table, or passage context.Only the question is emptied; the accompanying context remains unchanged.
- Integrated Gradients (IG): IG aggregates gradients along a path interpolating between the baseline and the input to attribute changes in prediction probability.This path-integral construction follows the prediction from the baseline state to the final input.
- Integrated Gradients (IG): The integrated-gradient definition computes each dimension’s attribution from the input–baseline difference and the gradient along that path.The paper introduces the formal per-dimension definition and identifies the gradient with respect to the corresponding input dimension.
- Integrated Gradients (IG): IG attributions sum to the prediction-probability difference between input and baseline, while uninfluential variables receive zero attribution.The paper also states that influential variables receive some attribution and that symmetric variables receive equal attributions.
- Integrated Gradients (IG): Question perturbations empirically validate IG: changing high-attribution terms alters responses, whereas changing low-attribution terms generally does not.The authors exploit this contrast when attacking models whose attributions favor generic words over contentful ones.
4 Visual Question Answering
The visual question answering analysis finds that the studied model often relies on generic question words and the image rather than informative terms. Attribution-guided perturbations expose this overstability and produce large accuracy drops.
- 4 Visual Question Answering: The analyzed Visual QA model answers image questions with 61.1% validation accuracy.The study uses the Kazemi and Elqursh network because it is easy to reproduce.
- 4.2 Observations: Attribution visualizations show that altering low-attribution question words can leave the model’s answer unchanged.Several semantically altered questions about the same image still receive the answer “very.”
- 4.2 Observations: The model’s highly attributed words are often generic terms, while informative words such as nouns frequently receive very low attribution.The authors interpret this as a weakness in the network’s logic and use it to design attacks.
- 4.3 Overstability test: One word, “color,” is sufficient for the model to achieve more than 50% of its final accuracy when other words are removed.The isolated vocabulary is formed from words that most frequently appear as top-attributed terms.
- 4.3 Overstability test: 44.3% of the model’s original accuracy remains even with empty questions, indicating substantial reliance on the image.The context image is retained while the question is emptied.
- Subject ablation attack: 75.6% of originally correct questions retain the same answer after replacing their subject with selected low-attribution nouns.The attack evaluates answer stability across replacements including “fits,” “childhood,” “copyrights,” and other nouns.
- Related work: The stability analysis extends earlier fragment-based observations and uses model-specific attributions to create additional attacks.The authors also report that attribution insights improve attacks proposed by Jia and Liang (2017).
5 Question Answering over Tables
The tabular QA analysis studies Neural Programmer on WikiTableQuestions, attributing operation and column choices to question words and table-match features. It finds strong overstability and identifies generic-word and trigger-based reasoning patterns.
- 5.1 Task, model, and data: WikiTableQuestions contains 22,033 questions over 2,108 Wikipedia tables, and Neural Programmer achieves 33.5% validation accuracy.Neural Programmer is weakly supervised from final answers and translates questions into four operator and column selections.
- 5.1 Task, model, and data: Integrated Gradients attributes Neural Programmer’s operator and column selections to question words, preprocessing tokens, and column-selection priors.The baseline uses an empty question and zero column-selection-prior vectors.
- 5.2 Observations: The attribution alignment reveals that “first” is selected for a superlative question and that “gold” triggers this operator.The visualization includes question words, preprocessing tokens, priors, and operator or column selections against baseline counterparts.
- 5.3 Overstability test: Five words plus column-selection priors provide more than 50% of Neural Programmer’s final validation accuracy.The words are “many,” “number,” “tm token,” “after,” and “total.”
5.4 Table-specific default programs
Neural Programmer can produce table-specific default programs whose operator choices depend substantially on the table rather than the question. Attribution analysis reveals this table-driven behavior and shows how it can yield incorrect answers.
- Table-specific default programs: 36.9% of selected operators match their table-default counterparts when the question is empty.These default programs are generated from the table alone.
- Table-specific default programs: Table 2 attributes default-program operator and column selections to frequently occurring column names across validation tables.The analysis uses IG to identify which column names are associated with these selections.
- Table-specific default programs: The “reset, prev, max, print” default program favors medal-related columns that often contain a “total” row.If a table lacks that row, the model may produce an incorrect answer.
- Table-specific default programs: The attacks described next exploit attribution-revealed weaknesses by adding or dropping content-free words from questions.This connects the default-program analysis to the subsequent adversarial attacks.
5.5 Attacks
Neural Programmer is highly sensitive to content-free phrases and stop words because some irrelevant words trigger operator selections. Attribution-guided trigger phrases cause much larger accuracy losses than low-attribution phrases.
- Question concatenation attacks: The attack phrases are manually constructed from irrelevant trigger words for operator selections and added as prefixes or suffixes.The phrases are checked to ensure they are content-free.
- Question concatenation attacks: 33.5% to 3.3%: four attribution-guided content-free phrases cause a large drop in Neural Programmer accuracy.The first two phrases alone reduce accuracy to 5.6%.
- Question concatenation attacks: Low-attribution non-trigger phrases reduce accuracy only from 33.5% to 27.1% when combined.Their smaller effect contrasts with phrases built from operator triggers.
- Question concatenation attacks: The word “at” can trigger the “next” operator, so deleting it from “what ethnicity is at the top?” changes the selected operation and answer.This illustrates how stop-word dependence can alter program execution.
- Question concatenation attacks: 33.5% to 28.5%: dropping manually selected stop words lowers validation accuracy.The paper notes that users may omit stop words in concise real-world queries.
Row reordering attacks
Neural Programmer relies on table row ordering, so rearranging rows exposes a substantial robustness gap. Training with perturbed tables is presented as a possible way to improve robustness and generalization.
- Row reordering attacks: For “which nation earned the most gold medals,” selecting “first” succeeds because the table is sorted by gold values.The “prev” operator also excludes the last row, exploiting table structure.
- Row reordering attacks: 33.5% to 23%: Neural Programmer’s accuracy falls on perturbed WikiTableQuestions tables without retraining.The perturbation tests sensitivity to row-order artifacts.
- Row reordering attacks: Question answering systems should not rely on row ordering in tables.The paper evaluates this bias using a perturbed validation dataset.
- Row reordering attacks: Training against perturbed tables is proposed as one approach to making the network robust to row-reordering attacks.The paper also suggests this may help the model generalize better.
- Row reordering attacks: A strongly supervised WikiTableQuestions model reportedly gains 7% final accuracy from training with perturbed tables.This result is attributed to Mudrakarta et al. (2018).
6 Reading Comprehension
In reading comprehension, attribution patterns predict when ADDSENT adversarial sentences will fool the model. Attacks are more effective when they preserve or target question words that the model treats as important.
- 6 Reading Comprehension: The evaluated reading-comprehension model is Yu et al. (2018), on a task that identifies an answer span from a context paragraph.The analyzed SQuAD model has a reported state-of-the-art F1 score of 84.6.
- 6 Reading Comprehension: ADDSENT appends answer-like sentences to paragraphs without changing the ground-truth answer.The attack targets reading-comprehension systems.
- 6.1 Analyzing adversarial examples: An attack can succeed when a low-attribution contentful question word is modified or when a question word absent from the context is introduced in the added sentence.The paper gives examples involving “Super” and “hotel.”
- 6.1 Analyzing adversarial examples: An attack can fail when a highly attributed contentful question word is missing from the added sentence, as with “Australia.”Retaining that word in the adversarial sentence changes the model’s prediction.
- 6.3 Predicting the effectiveness of attacks: 63% failed examples occur when a highly attributed noun or adjective is absent from the adversarial sentence, versus 40% otherwise.Across 1000 instances, 508 attacks fool the model and 492 do not.
- 6.2 Analyzing adversarial examples: Preserving nouns with high attributions during ADDSENT construction turns some previously unsuccessful attacks into successful ones.Table 4 presents examples of this modification.
7 Conclusion
Attribution analysis revealed weaknesses in three question answering models more effectively than conventional validation-set methods. The authors argue that attribution-based workflows can improve model-quality iteration and expose unsafe under-reliance on important question terms.
- Attributions identified weaknesses in three question answering models more effectively than conventional validation-set methods.
- Attribution-based workflows may help developers iterate on model quality more effectively.
- Under-reliance on important question terms is unsafe, even when the paper’s attacks appear unrealistic.
A Supplementary Material
The supplementary material documents adversarial weaknesses in Neural Programmer, including stop-word sensitivity, irrelevant operator triggers, and table-driven operator biases. These examples show how specific words or table properties can influence predictions inappropriately.
- The listed stop words include common function words such as “and,” “the,” “to,” and “of.”
- Neural Programmer’s accuracy falls from 33.5% to 28.5% when stop words are deleted from validation questions.
- Several Neural Programmer operator triggers are irrelevant; for example, “many” triggers “prev” despite lacking semantic relevance.
- Column attributions indicate that Neural Programmer is predisposed to select certain operators based solely on the table.