PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Attention for Vision-Language Tasks

Xu Yang, Hanwang Zhang, Guojun Qi, Jianfei Cai

arXiv:2103.03493v1cs.CV

TL;DR

Existing attention-based vision-language models can learn spurious correlations from hidden confounders, weakening causal input-target relationships and generalization. CATT uses front-door adjustment through In-Sample and Cross-Sample attention while preserving Q-K-V operations. It consistently improves several vision-language models, including making lighter LXMERT competitive with or better than UNITER on reported benchmarks.

  • Problem

    Unsupervised attention can be misled by dataset bias and hidden confounding, so modeling P(Y|X) may fail to identify the true causal effect from input X to target Y.

  • Method

    CATT applies front-door adjustment with In-Sample and Cross-Sample attention modules that retain Q-K-V operations and incorporate other samples to mimic intervention.

  • Results

    CATT achieves consistent improvements across Bottom-Up Top-Down LSTM, Transformer, and LXMERT, with LXMERT+CATT reaching 73.04% versus UNITER’s 72.91% on VQA2.0 test-std.

  • Takeaways & Limitations

    CATT can replace conventional attention modules and has reported potential for large-scale vision-language pre-training.

Abstract

from arXiv · show

We present a novel attention mechanism: Causal Attention (CATT), to remove the ever-elusive confounding effect in existing attention-based vision-language models. This effect causes harmful bias that misleads the attention module to focus on the spurious correlations in training data, damaging the model generalization. As the confounder is unobserved in general, we use the front-door adjustment to realize the causal intervention, which does not require any knowledge on the confounder. Specifically, CATT is implemented as a combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention (CS-ATT), where the latter forcibly brings other samples into every IS-ATT, mimicking the causal intervention. CATT abides by the Q-K-V convention and hence can replace any attention module such as top-down attention and self-attention in Transformers. CATT improves various popular attention-based vision-language models by considerable margins. In particular, we show that CATT has great potential in large-scale pre-training, e.g., it can promote the lighter LXMERT~\cite{tan2019lxmert}, which uses fewer data and less computational power, comparable to the heavier UNITER~\cite{chen2020uniter}. Code is published in \url{https://github.com/yangxuntu/catt}.

1. Introduction

Attention mechanisms selectively transform multimodal features but can be misled by dataset bias and hidden confounding. CATT applies front-door adjustment through a Q-K-V-compatible design to mitigate this effect and improves several vision-language models.

  • Attention background: Attention computes normalized query-key weights over feature values, producing a selective feature representation.The mechanism follows the common Q-K-V convention.
  • Attention background: Self-attention enriches local visual features with global relationships, while top-down attention enriches language context with cross-domain relationships.Both forms can be combined and stacked in multimodal networks.
  • Dataset bias: Unsupervised attention weights can follow dataset bias, such as inferring “riding” from person-horse correlations while ignoring carriage evidence.The resulting mismatch can hurt predictions on images with different relationships.
  • Dataset bias: A hidden confounder creates a backdoor path between input and target, so fitting P(Y|X) cannot identify the true causal effect even with more training data.The issue becomes especially relevant when confounder distributions differ between training and testing.
  • CATT: CATT uses front-door adjustment without requiring an observed confounder, combining In-Sample and Cross-Sample attention while retaining Q-K-V operations.This design allows CATT to replace conventional attention modules, including top-down attention and Transformer self-attention.
  • CATT: CATT produces consistent improvements across Bottom-Up Top-Down LSTM, Transformer, and LXMERT models.On VQA2.0, LXMERT+CATT reaches 73.04% versus 72.91% for UNITER on the test-std split.

2. Related Work

Prior work commonly formulates top-down and self-attention as Q-K-V operations, while causal-inference methods address dataset bias but often assume observable or domain-specific confounders. CATT is presented as a causal alternative for mitigating hidden confounding in attention.

  • Attention Mechanism: Top-down and self-attention mechanisms can be summarized using the query, key, and value operation.This formulation also extends beyond vision-language models to pure vision tasks.
  • Attention Mechanism: Because attention weights are unsupervised, hidden confounders in datasets can mislead the mechanism.The paper motivates CATT as a response to this confounding effect.
  • Causal Inference: Many causal bias-removal methods assume that confounders are observable or domain-specific.This assumption limits their direct generality across domains.

3. Causal Attention

Causal Attention decomposes attention into in-sample selection and cross-sample intervention to estimate causal effects while mitigating confounding bias. It combines these components within Q-K-V attention modules and can be integrated into stacked Transformer and LXMERT architectures.

  • 3.1. Attention in the Front-Door Causal Graph: The front-door formulation separates attention into a selector that chooses mediator knowledge Z from input X and a predictor that uses Z to predict Y.In VQA, the selector chooses image regions using the question before prediction.
  • 3.1. Attention in the Front-Door Causal Graph: Conventional attention can learn spurious correlations because unsupervised selection permits the backdoor path Z ←X ↔Y to influence prediction.The desired causal relation is Z →Y, rather than the dataset-driven correlation.
  • 3.1. Attention in the Front-Door Causal Graph: Cross-Sample Sampling blocks this backdoor path by stratifying inputs and combining features from other samples to approximate physical intervention.For example, pairing a person region with objects other than a snowboard discourages reliance on the dominant “man-with-snowboard” association.
  • 3.1. Attention in the Front-Door Causal Graph: CATT replaces the expensive sampling computation with Normalized Weighted Geometric Mean approximation, absorbing IS-Sampling and CS-Sampling into feature-level inputs.This reduces the need to forward every sampled combination through the network.
  • 3.2. In-Sample and Cross-Sample Attentions: IS-ATT applies Q-K-V operations to the current sample, while CS-ATT uses queries from f(X) and keys and values drawn from other samples.The two attentions estimate IS-Sampling and CS-Sampling, respectively; CS-ATT can use a K-means-compressed global dictionary from the training set.
  • 3.3. CATT in Stacked Attention Networks: A causal attention module concatenates the IS-ATT and CS-ATT outputs for prediction, and CATT can replace attention layers in stacked Transformer and LXMERT networks.The illustrated Transformer uses visual and language components, while LXMERT uses visual, language, and visiolinguistic modules with paired IS-ATT and CS-ATT paths.

4. Experiments

CATT is evaluated by replacing conventional attention in captioning, VQA, and vision-language pre-training models. Across these settings, CATT improves performance, reduces measured bias, and enables lighter LXMERT variants to approach or exceed heavier pre-training systems.

  • Experimental Setup: CATT is evaluated in BUTD LSTM, Transformer, and LXMERT models across image captioning, VQA, GQA, and NLVR2.The experiments use MS COCO, VQA2.0, and a large-scale pre-training dataset containing 180K images and 9.18M image-sentence pairs.
  • Image Captioning: 3.0-point and 3.2-point CIDEr-D improvements are achieved by BUTD+CATT and Transformer+CATT over their respective baselines.These CATT-equipped captioners also exceed certain state-of-the-art captioners using more complex techniques.
  • Visual Question Answering: CATT consistently improves BUTD and Transformer VQA accuracy across the reported validation and test splits.The deconfounded models also outperform certain state-of-the-art models that surpass the original BUTD and Transformer systems.
  • Visual Question Answering: 4.75-point and 2.75-point gains in number-question accuracy are reported for BUTD+CATT and Transformer+CATT, respectively.Transformer+CATT reaches 53.45 versus LXMERT’s 52.63 on number accuracy, while Figure 7 illustrates corrected attention to relevant regions.
  • Vision-Language Pre-training: 0.86-, 1.23-, and 1.6-point improvements are reported for LXMERT†+CATT on VQA2.0, GQA, and NLVR2, respectively.LXMERT†+CATT uses fewer GPU hours and less pre-training data than UNITER while achieving higher VQA2.0 and NLVR2 performances in the reported comparisons.
  • Ablation Studies: K-means initialization, parameter sharing, and larger dictionaries improve CATT ablation performance in the evaluated Transformer and LXMERT architectures.For LXMERT+CATT, the reported VQA comparisons are 70.40 versus 69.81 and 70.40 versus 70.05 for the tested strategies.

5. Conclusion

The paper analyzes attention through causal inference and proposes CATT, a front-door-adjustment mechanism that mitigates confounding bias across vision-language models.

  • Causal inference explains why attention can be misled by dataset bias and attend to unsuitable regions.
  • CATT uses In-Sample and Cross-Sample attention to estimate the samplings required by front-door adjustment.Both attention networks follow Q-K-V operations.
  • CATT improves various popular attention-based vision-language models by considerable margins.The paper highlights its potential for large-scale pre-training by promoting a light VLP model comparable to a heavy one.

6. Causal Preliminaries

The causal preliminaries define confounders and explain how backdoor and front-door adjustments estimate causal effects by blocking appropriate paths.

  • A confounder is a common cause of two variables that induces spurious correlation between them.In the example, C causes both X and Y.
  • Deconfounding X and Y requires blocking every backdoor path between them.For X ←C →Y, intervening on C blocks the path.
  • Chain and confounding junctions are blocked by intervening on their middle variables, whereas collider junctions are naturally blocked without intervention.
  • Backdoor adjustment estimates P(Y |do(X)) by averaging causal effects across confounder strata.It requires knowledge of the confounder to form those strata.
  • Front-door adjustment avoids the need to know the hidden confounder and calculates the causal effect through a mediator in a front-door structural causal model.Its derivation chains partially causal effects and blocks the relevant paths through collider structure and intervention on X.

7. Formula Derivations

The derivation approximates front-door causal adjustment with normalized weighted geometric means, yielding two attention samplings that are implemented through Q-K-V operations.

  • NWGM approximates an expectation by a weighted geometric mean when a predictive function has an exponential form.The approximation is used in deep networks whose final layer is Softmax.
  • The predictive distribution is parameterized as P(Y |X, Z) = Softmax[g(X, Z)] and is approximated using sampled representations.
  • The front-door approximation combines In-Sample Sampling E[Z|X][Z] with Cross-Sample Sampling E[X][X].Softmax normalizes the resulting exponential terms.
  • The sampled vectors are packed as ˆX and ˆZ, giving P(Y |do(X)) ≈Softmax[g(ˆZ, ˆX)].
  • In-Sample and Cross-Sample sampling each use a query set in a Q-K-V operation, with QI = h(X) and QC = f(X).A query remains necessary for Cross-Sample Sampling so its output does not collapse to a fixed vector for each input.

8. More Results

Additional experiments report strong online captioning performance and qualitative evidence that CATT helps models confront dataset biases.

  • A single Transformer+CATT achieves higher performance than other state-of-the-art models on the online MS COCO split.
  • CATT enables BUTD to generate the correct gender in an image despite the spurious training association between “woman” and “kitchen”.The qualitative examples cover both image captioning and VQA.

9. Implementation Details

CATT was integrated into both BUTD and Transformer architectures for image captioning and VQA, with task-specific attention inputs, training objectives, and optimization settings. Transformer CATT shared IS-ATT and CS-ATT parameters, adding trainable global image and word dictionaries beyond the original Transformer.

  • BUTD + CATT: CATT was deployed with BUTD for image captioning and VQA using one causal attention module.IS-ATT matched BUTD’s original attention module, while CS-ATT followed the paper’s cross-sample formulation.
  • BUTD + CATT: In BUTD, IS-ATT used current-image RoI features and sentence context, while CS-ATT used a 1000-cluster visual global dictionary.The dictionary was initialized with K-means over training-set RoI features.
  • BUTD + CATT: BUTD image captioning used cross-entropy and self-critical reward training, whereas VQA used binary cross-entropy with AdaDelta.Captioning training lasted 35 and 65 epochs for the two objectives; VQA training lasted 30 epochs.
  • Transformer + CATT: CATT was also integrated into Transformer architectures for image captioning and VQA, with IS-ATT and CS-ATT parameters shared within each CATT.Parameter sharing kept their outputs in the same feature space.
  • Transformer + CATT: Compared with the original Transformer, CATT added trainable global image and word embedding dictionaries as the additional parameters.The implementation otherwise retained the Transformer’s existing parameterization described in the passage.
  • Transformer + CATT: Transformer image captioning used sequential cross-entropy and self-critical reward training, while VQA used binary cross-entropy with Adam.Captioning used batch size 10; VQA training lasted 13 epochs.
Loading 2103.03493v1…