PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Causal Principles for Improving Visual Dialog

Jiaxin Qi, Yulei Niu, Jianqiang Huang, Hanwang Zhang

arXiv:1911.10496v3cs.CVcs.CL

TL;DR

VisDial models commonly rely on associations among image, history, question, and answer, but the paper argues that overlooked causal structure makes this treatment inadequate. It introduces two model-agnostic causal principles and intervention-based training, yielding substantial gains across representative baselines and leaderboard performance. The main scope boundary is that one question-type approximation relies on a limited observed prefix of the question, while the confounder itself is unobserved.

  • Problem

    Existing VisDial approaches overlook two causalities: a harmful direct history-to-answer shortcut and an unobserved confounder linking history, question, and answer.

  • Method

    The paper removes H →A and uses causal intervention P(A|do(I, H, Q)) with approximations to address the unobserved confounder.

  • Results

    Absolute NDCG gains reached 16.42% for LF, 15.01% for HCIAE, 15.41% for CoAtt, and 16.14% for RvA, while ensembles exceeded the 2019 winning performance by 0.2% and 0.9%.

  • Takeaways & Limitations

    The two principles are model-agnostic and can improve existing VisDial methods across ablative conditions, with additive effectiveness when combined.

  • Takeaways & Limitations

    The question-type approximation uses a small prefix of the question despite the backdoor-adjustment assumption that p(u|H) cannot condition on Q.

Abstract

from arXiv · show

This paper unravels the design tricks adopted by us, the champion team MReaL-BDAI, for Visual Dialog Challenge 2019: two causal principles for improving Visual Dialog (VisDial). By "improving", we mean that they can promote almost every existing VisDial model to the state-of-the-art performance on the leader-board. Such a major improvement is only due to our careful inspection on the causality behind the model and data, finding that the community has overlooked two causalities in VisDial. Intuitively, Principle 1 suggests: we should remove the direct input of the dialog history to the answer model, otherwise a harmful shortcut bias will be introduced; Principle 2 says: there is an unobserved confounder for history, question, and answer, leading to spurious correlations from training data. In particular, to remove the confounder suggested in Principle 2, we propose several causal intervention algorithms, which make the training fundamentally different from the traditional likelihood estimation. Note that the two principles are model-agnostic, so they are applicable in any VisDial model. The code is available at https://github.com/simpleshinobu/visdial-principles.

1. Introduction

The paper argues that VisDial differs fundamentally from VQA because dialog history and user preferences create overlooked causal structures. It proposes two model-agnostic principles—removing the direct history-to-answer link and addressing an unobserved confounder—with large gains across baseline models.

  • Motivation: VisDial is not simply VQA with history because its dialog data-generation process introduces distinct causal relationships.The paper contrasts conventional multimodal heuristics with causal inference over cause-effect structure.
  • Causal principles: Principle 1 deletes the direct H →A link so history informs questions without directly biasing answer prediction.The paper identifies H →Q as useful for resolving co-references while treating H →A as an undesirable shortcut.
  • Causal principles: Principle 2 adds an unobserved user-preference confounder U connected to history, question, and answer.The proposed graph adds U ←H, U →Q, and U →A, capturing spurious paths such as Q ←U →A.
  • Causal intervention: The proposed intervention P(A|do(I, H, Q)) cuts confounding paths and differs fundamentally from passive likelihood P(A|I, H, Q).Because U is unobserved after dataset construction, the paper develops approximations for the intervention.
  • Results: Absolute NDCG gains reached 16.42% for LF, 15.01% for HCIAE, 15.41% for CoAtt, and 16.14% for RvA.On the official test-std server, ensembles also exceeded the team’s 2019 winning performance by 0.2% and 0.9%.

2. Related Work

VisDial is an interactive, free-form vision-language task in which users jointly shape the dialog data. Existing methods mainly use encoder-decoder architectures that differ in how they process history, while the paper identifies a shared causal oversight.

  • Visual Dialog: VisDial uses live chat in which the questioner cannot see the image, asks open-ended questions, and the answerer gives free-form responses.The paper applies this setting rather than the goal-driven GuessWhat?! protocol.
  • Visual Dialog: User participation in data collection is a key difference between VisDial and many other vision-language tasks.The interactive protocol makes human roles central to the dataset-generation process.
  • Related approaches: Existing VisDial approaches use encoder-decoder frameworks and process history holistically, hierarchically, or recursively.Examples include holistic HACAN, DAN, and CorefNMN; hierarchical HRE; and recursive RvA.
  • Related approaches: These history-processing approaches overlook the paper’s claim that history should not be directly fed into the answer model.The paper evaluates LF, HCIAE, CoAtt, and RvA as representative baselines.

3. Visual Dialog in Causal Graph

The paper represents Visual Dialog with a causal graph whose nodes encode variables and whose arrows encode causal relationships. Existing encoder-decoder models transform image, question, and history features before decoding candidate answers.

  • 3.2. Encoder-Decoder as Causal Graph: The paper formalizes the popular encoder-decoder framework as a baseline causal graph before introducing alternative causal principles.The formal task definition and causal graph provide the basis for analyzing existing methods.
  • 3.1. Visual Dialog Settings: Visual Dialog gives a model an image, dialog history, current question, and 100 candidate answers to rank.Evaluation uses NDCG, which rewards placing human-relevant candidates higher rather than selecting only one ground-truth answer.
  • 3.2. Encoder-Decoder as Causal Graph: A causal graph is a directed acyclic graph whose nodes are variables and whose arrows represent causal relationships between them.An arrow A →B means that A is the cause and B is the effect.
  • 3.2. Encoder-Decoder as Causal Graph: Existing models encode image, question, and history features, use attention to obtain visual knowledge, then fuse features in a decoder to generate answers.The baseline graph represents these transformations as paths including I →V, Q →V, H →Q →V, and direct inputs H →A, Q →A, and V →A.

4. Two Causal Principles

The paper identifies two causal problems in Visual Dialog: direct history-to-answer input creates shortcut bias, while an unobserved user preference confounds history, questions, and answers. It proposes intervention through do-calculus to estimate the answer effect of the inputs.

  • 4.1. Principle 1: The double-blind analogy motivates blocking a direct link that can transmit undesirable bias instead of relying on the intended mediated path.In the analogy, author anonymity blocks researcher identity from directly affecting acceptance; in VisDial, question processing should mediate history’s influence on answers.
  • 4.1. Principle 1: Principle 1 removes the direct history-to-answer path because history should help resolve question coreferences, while the question mediates answer prediction.Direct history input can make answers match history words even when those words are irrelevant to the question.
  • 4.2. Principle 2: Principle 2 identifies an unobserved user preference as a confounder that creates backdoor paths from history and question to answer.Annotators see dialog history while producing answers, so nuanced preferences can causally affect both questions and answers.
  • 4.3. do-calculus: Do-calculus intervenes on the inputs, cuts incoming confounding arrows, and uses P(A|do(Q, H, I)) rather than passive likelihood P(A|Q, H, I).The resulting objective applies backdoor adjustment by summing P(A|Q, H, I, u)P(u|H).
  • 4.3. do-calculus: Because the confounder is unobserved, the paper next develops implementations that approximate the intervention rather than sampling the confounder directly.The paper states that the detailed derivation and proof are provided in supplementary materials.

5. Improved Visual Dialog Models

The paper implements confounder removal with candidate-based approximations, answer-score weighting, and a learned hidden dictionary. These methods approximate the interventional answer distribution while retaining the causal principles.

  • 5.1. Question Type: The first approximation replaces unobserved confounder samples with answer candidates observed from users’ minds during dataset collection.It uses candidate probabilities conditioned on question type to approximate the history-dependent prior.
  • 5.1. Question Type: Question-type sampling scores candidates by their frequency under the extracted question type, then uses those scores in the approximated intervention.The approximation is considered reasonable because question type is only a small part of the question, although it technically conditions the prior on Q.
  • 5.2. Answer Score Sampling: Answer Score Sampling uses normalized human-annotated NDCG scores to approximate the preference prior P(ai|H).The method uses dense annotations, where scores directly reveal human preference for candidate ai in context H.
  • 5.2. Answer Score Sampling: Three losses implement the answer-score approximation: Weighted Softmax, Binary Sigmoid, and a ranking loss based on candidate relevance.The paper emphasizes that these losses derive from the causal principle rather than from directly regressing to ground-truth NDCG.
  • 5.3. Hidden Dictionary Learning: Hidden Dictionary Learning models the unobserved confounder with an N × d dictionary and approximates its expectation through attention over dictionary entries.The answer distribution uses a softmax similarity function over candidate embeddings, confounder representations, and the joint question-image-history embedding.
  • 5.3. Hidden Dictionary Learning: The hidden-dictionary implementation uses an efficient NWGM approximation to move the confounder expectation inside the softmax.A dot-product attention mechanism computes the history-conditioned expected dictionary representation.

6. Experiments

Experiments on VisDial v1.0 show that the two principles improve diverse baseline models, reach top leaderboard ranks, and reduce history-based shortcuts while producing more reasonable answer rankings.

  • Experimental setup: The principles were evaluated on VisDial v1.0 using four representative baseline models and validation-set NDCG comparisons.The experiments include LF, HCIAE, CoAtt, and RvA, with implementations and ablations for P1 and P2.
  • P2 implementation: P2 implementations improve base-model performance by at most 11.75%, with hidden dictionary learning providing the largest boost and R3 ranking loss performing best.The compared P2 implementations include question type, answer-score sampling, and hidden dictionary learning; R3 is preferred because it satisfies VisDial’s ranking property.
  • Ablation results: Applying P1 and P2 improves all four models under every ablation, and combining the principles performs best because their effects are additive.The reported models are LF, HCIAE, CoAtt, and RvA, with conditions for P1, P2, and P1+P2.
  • Qualitative analysis: P1 reduces history word matching from approximately 5,200 to 4,800 validation-set cases among top-10 candidates, alleviating shortcut choices.The qualitative examples describe irrelevant history words influencing baseline rankings and report reduced shortcut behavior after P1.
  • Qualitative analysis: P2 ranks semantically suitable answers such as “yes, he is,” “yes he is,” and “I think so” together with the ground-truth “yes,” improving NDCG.The examples contrast this behavior with the baseline’s focus on exact ground-truth answers such as “no” or “yes.”

7. Conclusions

The paper proposes two model-agnostic causal principles for Visual Dialog and supports them with implementations and experiments showing major improvements across existing methods and evaluation servers.

  • Conclusions: The paper proposes two causal principles for improving Visual Dialog that are model-agnostic and applicable to almost all existing methods.The principles arise from causal analysis of the nature of Visual Dialog, which the authors state has been overlooked by the community.
  • Conclusions: Implementations of the principles were evaluated through extensive experiments on the official VisDial dataset and online evaluation servers.The conclusion describes the resulting performance as promising and says the experiments demonstrate the principles’ effectiveness.
  • Conclusions: The authors plan to apply causal thinking to other potential causalities in embodied question answering and conversational visual dialog.This is presented as future work following the reported VisDial results.
Loading 1911.10496v3…