PaperlayerView exact revision on arXiv

Source-linked AI summary

Deconfounded Image Captioning: A Causal Retrospect

Xu Yang, Hanwang Zhang, Jianfei Cai

arXiv:2003.03923v2cs.CV

TL;DR

Image captioners can reflect dataset bias, but prior work lacks a principled account of why they learn it. The paper uses causal inference to analyze this problem and proposes DICv1.0, which improves two prevailing captioning models on MS COCO.

  • Problem

    Prior work lacks a principled analysis of why vision-language systems, including image captioners, easily learn dataset bias.

    Source: §1 · p. 1

  • Method

    The paper analyzes image captioning through causal inference and proposes DICv1.0 based on backdoor and front-door adjustments.

    Source: §1 · pp. 1–2

  • Results

    126.4 to 129.5 CIDEr-D for Up-Down and 128.7 to 131.1 for AoANet; AoANet also achieved 128.4 CIDEr c40 on the MS COCO Caption test server.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that mediators and additional resources alone are insufficient for high-quality captions without identifying hidden confounders and applying backdoor and front-door adjustments.

    Source: §5.2.1 · p. 9

    Backdoor adjustment requires advance knowledge of confounders, which can be complex and difficult to identify or disentangle in real situations.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community. Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias. In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias. DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split of the challenging MS COCO dataset, respectively. Interestingly, DICv1.0 is a natural derivation from our causal retrospect, which opens promising directions for image captioning.

1 INTRODUCTION

The introduction frames dataset bias as a reason benchmark performance can be untrustworthy and presents Deconfounded Image Captioning (DIC) as a causal analysis and model-design response. It attributes caption bias to dataset concepts confounding image features and language, and proposes DICv1.0 using backdoor and front-door adjustment.

1 INTRODUCTION

  • Dataset bias can produce gender-discriminating captions, hallucinated objects or actions, and image-independent VQA answers, while collecting a perfectly balanced dataset is impractical.The introduction argues that this motivates shifting from collecting new datasets toward designing less biased models.Source: §1 · p. 1
  • Frequent concepts D affect both image features X and caption language L, encouraging captioners to exploit spurious concept-word co-occurrences instead of learning the true X →L relationship.For example, recognizing IDBoard can lead a captioner to choose “man” because of its higher training-set co-occurrence with that concept.Source: §1 · p. 2
  • DIC targets P(L|do(X)) rather than observational P(L|X), using backdoor and front-door adjustments,, to block the confounding path and recover the causal effect.The do-operator cuts the X ←D link, blocking the backdoor path X ←D →L.Source: §1 · pp. 1–2
  • Up-Down improves from 126.4 to 129.5 CIDEr-D and AoANet from 128.7 to 131.1 CIDEr-D; AoANet reaches 128.4 CIDEr c40.The paper reports that the AoANet result was submitted to the MS COCO Caption test server.Source: §1 · p. 2
  • The paper retrospectively analyzes major captioners from the six-year-old image-captioning community through a causal lens, using that retrospect to motivate DICv1.0 and future directions.The authors also present image captioning as a case study because of its long history and comparatively simple cross-modal objective.Source: §1 · pp. 1–2

2 PRELIMINARIES: DECONFOUNDING

The section frames deconfounding as learning captions from image features rather than spurious correlations, and reviews backdoor and front-door adjustments as ways to estimate causal effects. It explains how each approach addresses dataset bias and where front-door adjustment is useful.

2.1 The Backdoor Adjustment

  • Frequent concepts such as objects can confound captioning: models may infer attributes or relations from co-occurrence rather than reasoning from the image.For example, after recognizing an apple, a captioner may predict “red” from dataset frequency without seeing the apple’s actual color.Source: §2.1 · p. 3
  • Backdoor adjustment averages across confounder concepts, encouraging captions to connect image features with labels instead of relying on a single preferred concept.In the digit example, intervening on position makes left and right positions count regardless of their frequency in the data.Source: §2.1 · p. 3

2.2 The Front-door Adjustment

  • Backdoor adjustment requires identifying and splitting confounders, which is difficult when captioning biases involve concept combinations or unknown pretraining data.A relation word may depend on both subject and object concepts, while researchers may not know which concepts were frequent in the pretraining data.Source: §2.2 · p. 3
  • Front-door adjustment inserts a mediator Z between image features X and caption L, chaining their causal effects; in the digit example, Z represents shape.The paper notes that visual attention over features from different positions has also been used as a mediator in captioners.Source: §2.2 · pp. 3–4
  • A mediator alone may still transmit position bias, but cutting Z←X blocks the relevant backdoor path and encourages the classifier and extractor to learn causal effects.The classifier can learn that “circle” causes “0” as representations vary while the shape knowledge remains fixed, encouraging the extractor to recover the shape from images.Source: §2.2 · p. 4

3 RELATED WORK

The causal retrospect argues that common captioning strategies do not fully remove dataset confounding because training data and intermediate representations remain linked to biased concepts. The review distinguishes this challenge from prior causal methods and group-based captioning.

3.1 A Causal Retrospect

  • Large-scale training with Visual Genome, fine-grained annotations, or 3 million Conceptual Captions pairs boosts performance but cannot eliminate bias.The paper frames Show & Tell as the shift to neural encoder-decoder captioning; subsequent pretrained models narrow the vision-language gap while intensifying preferred-concept learning.Source: §3.1 · p. 4
  • Attention approximates backdoor and front-door adjustment, but averages only over concepts in the image and uses observational probabilities, so it does not fully deconfound captioning.Image-text matching supervision from the confounder also leaves captioners confounded.Source: §3.1 · pp. 4–5
  • Sentence-pattern captioners such as NBT and CNM remain confounded because dataset concepts affect both their patterns and generated captions.Since the dataset creates the causal link D → Z, front-door adjustment cannot be applied.Source: §3.1 · p. 5
  • Scene-graph captioners,,, use structured attention, but biased relations and parsers trained on VG link their structures to dataset concepts.Because D → Z persists, front-door adjustment cannot deconfound these models, which remain encouraged to learn spurious correlations,.Source: §3.1 · p. 5

3.2 Causal Inference in Computer Vision

  • Prior causal vision methods either deconfound with backdoor adjustment assuming an observable confounder,, or use Natural Indirect and Total Direct Effects,.Here the confounder is unobservable, making the problem more challenging and motivating use of both backdoor and front-door adjustments.Source: §3.2 · p. 5

3.3 Group Caption

  • Unlike group-based captioning,, DIC targets single-image captions without explicit image groups and measures average causal effect rather than separate relevance and diversity.Group-based methods use intra- and inter-group relevance and diversity, while DIC uses implicit groups of similar visual concepts or features.Source: §3.3 · pp. 5–6

4 DECONFOUNDED IMAGE CAPTIONING

The section derives DICv1.0 by choosing a mediator and addressing confounding, then develops expectation approximations and modules and integrates them into encoder-decoder captioners.

4.1 Choosing Z

  • DICv1.0 uses ConceptNet commonsense structures as mediator Z, rejecting spatial position and sentence pattern, and applies front-door adjustment for dataset confounder D and backdoor adjustment for observable vocabulary S.S confounds Z and L because it supplies both structure-search keywords and caption words, inducing additional vocabulary-related biases.Source: §4.1 · p. 6

4.2 Normalized Weighted Geometric Mean

  • The Normalized Weighted Geometric Mean approximation moves expectations through the Softmax and, for an FC layer, estimates P(L|do(X)) as Softmax of the expected keyword, visual, and semantic embeddings.The approximation allows the network to estimate the expectation with a single forward pass rather than sampling a huge number of outputs.Source: §4.2 · p. 6, §4.1 · p. 6

4.3 EXPT Modules for Expectations

  • EXPT estimates expectations by learning a basis dictionary X from training-image representations and approximating P(x|h) with a query-key operation, where h is query and X elements are keys.The dictionaries X, Z, and S contain 10,000, 9,590, and 1,342 elements, respectively.Source: §4.3 · p. 7

4.4 Network Architectures

  • DICv1.0 is integrated into Up-Down and AoANet with attention and EXPT modules, retaining each model’s distinct attention and embedding layers; both report less bias and better performance than their original models.The shared decoder uses visual features, prior-word embeddings, and prior outputs, with attention producing the image feature used by EXPT.Source: §4.4 · pp. 7–8

4.5 Training Objectives

  • Training uses cross-entropy on P(L*|do(X)) for 35 epochs, then an RL-based loss for 65 more epochs, with a sentence-level reward such as CIDEr-D.The models use Adam, and the learning rate is decayed by 0.8 every five epochs.Source: §4.5 · p. 8

5 EXPERIMENTS

Across MS COCO experiments, the authors report that DIC improves caption similarity and reduces measured bias; its variants achieve top CIDEr-D scores against the compared systems and receive better human consistency judgments.

5.1 Datasets

  • The experiments used the Karpathy and official online splits, and built ConceptNet dictionaries from 1,342 keywords and 9,590 selected semantic structures.Captions were capped at 16 words, with a 10,369-word vocabulary; the split sizes were 113,287/5,000/5,000 and 82,783/40,504/40,775 images for training/validation/test, respectively.Source: §5.1 · p. 8

5.2.1 Deconfounding Techniques

  • UD-DICv1.0 raised CIDEr-D from UD’s 125.3 to 128.8, while UD-BD/CC reached 127.5, supporting backdoor adjustment and showing similarity gains with a larger dictionary.Sequential improvements from UD-FD/Cor through UD-FD to UD-DICv1.0 supported deconfounding D and S; deconfounding S outperformed adding dictionary parameters.Source: Table 1 · p. 9, §5.2.1 · p. 9
  • UD-DICv1.0 had the lowest CHs&CHi and highest accuracy across four word-bias categories, though action bias remained difficult to reduce.For IDBed, UD accuracy was below 0.6 while UD-DICv1.0 approached 0.9; action-word accuracy for IDHorse remained below 0.4 for three models, and language and selection biases complicate evaluation.Source: Table 1 · p. 9, §5.2.1 · pp. 9–10, Figure 6 · p. 9
  • Human judges rated UD-DICv1.0 captions more image-consistent than UD (70% vs. 12%) and UD-BD (61% vs. 14%), and found better word-region alignment than UD (68% vs. 17%).Its A@Quan was also higher than UD’s (0.57 vs. 0.46), which the authors interpret as evidence of better attention.Source: Table 2 · p. 9, Table 1 · p. 9, §5.2.1 · p. 10

5.2.2 Network Size

  • Adding LSTM layers alone did not explain UD-DICv1.0’s gains: its CIDEr-D was 128.8, while extra layers did not clearly reduce CHs or CHi, and changing dictionary sizes had marginal effects.The comparisons suggest the deconfounding approximations, rather than network size or dictionary size, account for most improvements, especially in bias reduction.Source: Table 3 · p. 10, §5.2.2 · p. 10

5.3 Comparisons with State of the Art

  • 128.8 CIDEr-D for UD-DICv1.0 and 131.1 for AoA-DICv1.0 were the best single-model scores under different training rewards on the Karpathy split.The comparison includes large-scale pre-training and advanced-attention captioners.Source: §5.3 · p. 11, Table 4 · p. 12
  • UD-DICv1.0 outperformed RFNet, CAVP, and LBPF, while AoA-DICv1.0 outperformed AoANet; UD-BD and UD-DICv1.0 also achieved higher BLEU and METEOR than GroupCap.The authors report competitive results against systems using substantially more pre-training data and argue that deconfounding is an effective alternative to more complex attention, sentence-pattern, or graph methods.Source: §5.3 · pp. 11–12, Table 4 · p. 12, Table 5 · p. 12
  • UD-DICv1.0 and AoA-DICv1.0 achieved the highest CIDEr-D c5 and c40 scores among compared captioners on the MS COCO online test set.The passage reports the ranking but gives no score values.Source: §5.3 · p. 12

6 CONCLUSIONS

The conclusion explains captioners’ dataset bias through confounding and presents causal adjustments as both a lens for prior work and the basis for DICv1.0. It identifies an assumption that may limit DICv1.0 and sketches a recursive path for future work.

6 CONCLUSIONS

  • Neglecting frequent confounding concepts can make captioners trained on observational P(L|X) collapse into dataset bias; backdoor and front-door adjustments instead use interventional P(L|do(X)) as the training target.The paper also finds that many advanced captioners approximate these adjustments.Source: §6 · p. 12
  • DICv1.0 derives from the causal retrospect and was validated by incorporating it into the prevailing Up-Down and AoANet models.The conclusion describes the method as an effective approach, but this passage supplies no performance values.Source: §6 · p. 12
  • DICv1.0 assumes S can be split into keywords for deconfounding, but if it cannot, the paper proposes reassigning S to D and Z to X and recursively deploying prior techniques.This would return to the paper’s starting point, with this work becoming part of “previous IC” in the recursive future.Source: §6 · p. 12
  • Table 6 reports single-method scores on the online MS COCO test server, separating training with CIDEr-D computed from 5 captions from training with the whole training set.The caption does not provide individual scores or establish which method performs best.Source: Table 6 · p. 12

Supplementary Material of Deconfounded Image … B FORMULA DERIVATIONS

The supplement outlines its coverage, then explains how causal graphs represent spurious backdoor paths and how controlling variables blocks some paths but not colliders. Its causal preliminaries define the reasoning needed to analyze the causal effect of X on L.

Supplementary Material of Deconfounded Image

A.1 Causal Graph

  • In a causal graph, arrows mark causal or anticausal links, and the backdoor path X ←D →L can spuriously correlate X and L.The graph also contains the causal path X →L.Source: §A.1 · p. 15
  • A backdoor path from X to L is any path that starts with an arrow into X; to estimate P(L|do(X)), block every such path.In Fig. A(a), blocking X ←D →L is sufficient.Source: §A.1 · p. 15
  • Backdoor paths can also occur between other variable pairs, including X and Z via X ←D →L ←Z and Z and L via Z ←X ←D →L.Source: §A.1 · p. 15

A.2 Blocking Path

  • In chain and confounding junctions, controlling the mediator or confounder blocks information flow between the endpoints.The chain is A →B →C, and the confounding junction is A ←B →C.Source: §A.2 · p. 15
  • In a collider A →B ←C, the endpoints are independent when B is unknown but become correlated when B is known, so the path is blocked by not controlling B.Source: §A.2 · pp. 15–16

B.1 Derivations of Eq. (17) and Eq. (21)

The section derives sentence-level P(L|do(X)) by chaining causal partial effects, then extends the derivation to word-level captioning conditioned on accumulated context H.

B.1 Derivations of Eq. (17) and Eq. (21)

  • Eq. (17) obtains sentence-level P(L|do(X)) by chaining P(Z|do(X)) and P(L|do(Z)).The derivation follows the front-door adjustment procedure.Source: §B.1 · p. 16
  • For P(L|do(Z)), controlling S and X blocks the backdoor paths Z ← S → L and Z ← X ← D → L; the paths between X and Z are already blocked by colliders.D is not accessible, so X is controlled to block the path through D.Source: §B.1 · p. 16
  • At the word level, Eq. (21) conditions on accumulated caption context H=h, which blocks paths through H and modifies the sentence-level derivation.Although S has no direct link from H and P(s|h) should be P(s), the experiments condition S on h for representation power; without this, the expected S degrades to a fixed vector.Source: §B.1 · p. 16

B.2 Derivations of Eq. (19) and Eq. (20)

The derivations use normalized weighted geometric mean (NWGM) to approximate expectations in captioning distributions. They apply the approximation to the sentence-level and word-level distributions, with EXPT modules estimating expectations and the sentence representation conditioned on the hidden state.

B.2 Derivations of Eq. (19) and Eq. (20)

  • NWGM approximates an expectation by the weighted geometric mean; for exponential functions, it absorbs the expectation into the exponential term.The paper applies this approximation to deep networks with a Softmax final layer.Source: §B.2 · p. 16
  • For Eq. (19), the sentence-level distribution is approximated with NWGM and normalized by Softmax so the resulting probabilities sum to 1.Because the distribution is only proportional to the exponential parameterization, the NWGM step is approximate.Source: §B.2 · p. 17
  • NWGM also approximates the word-level distribution, and EXPT modules estimate the expectations with S conditioned on h to avoid E[s][s] remaining constant during caption generation.The authors report that without conditioning S on h, E[s][s] has the same value throughout caption generation.Source: §B.2 · p. 17

C NETWORK ARCHITECTURE … C.3 Common Structure of the Decoder

Appendix C details the architectures of UD-DICv1.0 and AoA-DICv1.0, describing the EXPT and attention modules and the decoders’ shared structure and implementation choices.

C NETWORK ARCHITECTURE

  • This section details the network architectures of UD-DICv1.0 and AoA-DICv1.0, models proposed in Section 4.4 of the submitted manuscript.Source: C NETWORK ARCHITECTURE · p. 17

C.1 EXPT Module

  • The EXPT module’s operation is shown in Eq. (25), and its detailed structure is listed in Table A.Source: §C.1 · p. 17

C.2 ATT Module

  • UD-DICv1.0 uses Top-Down Attention, while AoA-DICv1.0 uses Attention on Attention as their respective ATT modules.The module details are given in Tables B and C, respectively.Source: §C.2 · p. 17
  • Self-attention forms M by concatenating eight heads and multiplying by WC, then computes ˆv as LeakyReLU(MLP(M)).The passage specifies the multihead combination and output operation.Source: §C.2 · p. 17
  • k ∈R1,000∗125 and WC ∈R1,000∗1,000 are trainable matrices, with dk = 1000/8 = 125.Source: §C.2 · p. 17

C.3 Common Structure of the Decoder

  • The two decoders share a common structure, with ATT set to Top-Down Attention or Attention on Attention and g(·) set to an LSTM or GLU layer.The common structure is given in Table D.Source: §C.3 · p. 17

D MORE RESULTS … E.1 Dataset

The supplementary sections examine how dataset co-occurrences can bias captions, how DICv1.0 behaves on a more balanced dataset and in attention examples, and why VATEX offers a case for studying video-captioning bias.

D.1 Effects of the Dataset Biases.

  • COCO exhibits long-tail word and visual-concept frequencies, gender-object co-occurrences, and action co-occurrences such as “person” and “horse” with “ride” more often than “stand with”.These patterns can lead captioners to infer frequent genders or actions from recognized objects; biases can also arise from combinations of three or more words.Source: §D.1 · p. 17
  • Because complex and multi-concept biases cannot be exhaustively enumerated or fully balanced, the authors create a relatively balanced set by equalizing gender-object co-occurrences.They delete captions where an object's co-occurrence with one gender exceeds the other's; 34,576 captions are removed, and the remainder train and test UD-DICv1.0 and UD.Source: §D.1 · pp. 17–18
  • On the balanced-data experiment, UD-DICv1.0* loses less CIDEr-D with fewer examples, has smaller A@Gen gains than UD*, and still outperforms UD* although neither model reaches 100% accuracy.The authors suggest ConceptNet knowledge may help when less data are available, while other factors—including nonvisual concepts and concept combinations—may leave residual gender bias.Source: §D.1 · p. 18

D.2 More attention comparisons

  • In an attention example, UD attends to a computer rather than a person and generates the incorrect gender, while UD-DICv1.0 attends to the person and generates “woman”.The comparison is presented as evidence that UD-DICv1.0 can improve attention quality.Source: §D.2 · p. 18

E.1 Dataset

  • VATEX is presented as a suitable bias study case because its action-classification-pretrained I3D features may encourage captioners to overuse action knowledge.The dataset has over 41,250 videos, 825,000 captions, and 600 human activities; the paper uses its official train/validation/test split.Source: §E.1 · p. 18
  • VATEX experiments use the official split of 25,991/3,000/6,000 videos for training, validation, and testing, respectively.The passage also notes that VATEX provides an official I3D visual feature extractor for comparison with other models.Source: §E.1 · p. 18

E.2 Experiments and Results

On VATEX, UD-DICv1.0 and AoA-DICv1.0 achieved higher CIDEr-D scores than HRN, ORG, and ORG+TRL. The reported results also show UD-DICv1.0 outperforming UD on CIDEr-D for both integral-data and gender-balanced training.

E.2 Experiments and Results

  • The VATEX comparisons used captioners with the same visual features and LSTM-based decoders, including UP-Down, VATEX, AoA, HRN, and ORG.Source: §E.2 · p. 18
  • 128.8 CIDEr-D for UD-DICv1.0 exceeded UD’s 125.3 on the integral dataset.On the gender-balanced dataset, UD-DICv1.0* scored 123.4 CIDEr-D versus 118.1 for UD*.Source: §E.2 · p. 19
  • UD-DICv1.0 and AoA-DICv1.0 achieved higher CIDEr-D scores on VATEX than HRN, ORG, and ORG+TRL.The paper attributes this comparison to deconfounding techniques in video captioning.Source: §E.2 · p. 19

Related papers

Loading 2003.03923v2…