PaperlayerView exact revision on arXiv

Source-linked AI summary

Deconfounded Image Captioning: A Causal Retrospect

Xu Yang, Hanwang Zhang, Jianfei Cai

arXiv:2003.03923v2cs.CV

TL;DR

Vision-language systems can collapse into dataset bias because frequently appearing concepts confound image features and captions, while existing analyses lack a principled explanation. The paper uses causal backdoor and front-door adjustments to develop DICv1.0, which strengthens prevailing captioners and reaches 131.1 CIDEr-D and 128.4 CIDEr c40 on MS COCO evaluations.

  • Problem

    Existing vision-language solutions lack a principled analysis of why image captioners collapse into dataset bias, which affects learned representations and captions.

  • Method

    DICv1.0 uses causal backdoor and front-door adjustments to analyze confounders and train captioners with interventional effects.

  • Results

    131.1 CIDEr-D and 128.4 CIDEr c40 were achieved by DICv1.0 on the reported MS COCO evaluations.

  • Takeaways & Limitations

    Causal retrospect identifies hidden confounders and motivates deconfounding models for alleviating dataset bias in image captioning.

  • Takeaways & Limitations

    Backdoor adjustment is difficult when confounders are complex or unavailable, and the chosen keyword vocabulary can introduce an additional confounder between mediator structure and captions.

Abstract

from arXiv · show

Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community. Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias. In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias. DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split of the challenging MS COCO dataset, respectively. Interestingly, DICv1.0 is a natural derivation from our causal retrospect, which opens promising directions for image captioning.

1 INTRODUCTION

The paper frames dataset bias as a causal problem in image captioning: confounding concepts can make models rely on spurious correlations rather than image features. It introduces Deconfounded Image Captioning (DIC), using causal inference to analyze and alleviate this problem.

  • Dataset bias produces gender discrimination, hallucinated content, and answers that can ignore image evidence in vision-language systems.
  • Collecting a perfectly balanced dataset is impractical because real-world data contains long-tailed, reporting, and language biases.
  • Frequently appearing visual concepts act as confounders by affecting both learned image representations and caption-word frequencies.
  • DIC applies the backdoor and front-door adjustments to replace observational captioning probability P(L|X) with interventional probability P(L|do(X)).
  • 131.1 CIDEr-D was achieved by AoA-DICv1.0, improving from 128.7, while the online split reached 128.4 CIDEr c40.

2 PRELIMINARIES: DECONFOUNDING

The preliminaries explain how observational training can exploit confounding correlations and introduce backdoor and front-door adjustments for learning causal image-to-caption effects. Because complex captioning confounders are difficult to identify, the front-door approach uses a mediator to divide and deconfound the process.

  • Position-biased digit classification illustrates how observational training favors the shortcut X ←P →L over the true effect X →L.The classifier may rely on whether black pixels appear on the left or right because that feature is easier to learn than digit shape.
  • The backdoor adjustment replaces P(p|X) with P(p), preserving less frequent position-feature combinations so the classifier can learn beyond position.
  • Training on P(L|X) can make captioners exploit frequently appearing concepts instead of reasoning about attributes and relations from images.For example, recognizing IDApple can encourage predicting “red” from its co-occurrence rather than visual evidence.
  • The backdoor adjustment is impractical when dataset confounders are complex, combinatorial, or unavailable after pre-training data are discarded.
  • The front-door adjustment inserts a mediator Z, decomposing the process into X →Z and Z →L and chaining P(Z|do(X)) with P(L|do(Z)).
  • For the mediator extractor, the backdoor path is blocked by a collider, whereas the mediator classifier requires cutting Z ←X to block its confounding path.

3 RELATED WORK

The causal retrospect reinterprets major image captioners through dataset bias and deconfounding. It shows why scaling, sentence patterns, and structured attention may still leave captioners vulnerable to spurious correlations.

  • The causal retrospect provides a causal perspective for understanding milestone image captioners and their contributions.
  • Large-Scale Training: Large-scale training approximates deconfounding by expanding data and label coverage, but does not fully remove dataset bias.
  • Sentence Patterns: Sentence-pattern models dynamically compose modules for different word patterns, resembling a front-door architecture at first glance.
  • Structured Attention: Structured-attention models transform learned scene graphs into sub-structure embeddings that guide caption generation.
  • Structured Attention: Scene-graph parsers trained on the same biased data as captioners create D → Z, preventing direct use of front-door adjustment and preserving spurious correlations.

4 DECONFOUNDED IMAGE CAPTIONING

This section derives DICv1.0 by selecting a mediator and combining front-door and backdoor adjustments for two confounders. The resulting intervention is implemented through an expected conditional distribution and a single-forward-pass approximation.

  • The section derives DICv1.0 by selecting a mediator, computing P(L|do(X)), addressing its deep-network challenges, and integrating it into an encoder-decoder.
  • Choosing Z: Spatial position covers only a small subset of dataset concepts, while sentence patterns are unavailable during caption generation, making both unsuitable mediators.
  • Choosing Z: DICv1.0 selects commonsense structures from ConceptNet as mediator Z, representing subject-predicate-object triplets with semantic embeddings.
  • Choosing Z: Caption-vocabulary keywords introduce S → Z and S → L, making S an additional confounder that induces related-word biases.
  • Deconfounding: DICv1.0 deconfounds D with front-door adjustment and observable vocabulary confounder S with backdoor adjustment.
  • Implementation: P(L|do(X)) is implemented as a Softmax network for P(L|s,x,z), with s, x, and z denoting keyword, visual, and semantic embeddings.
  • Implementation: The NWGM approximation moves expectations into the fully connected layer, yielding a single-pass estimate of the interventional distribution.

s¯ s¯ LSTM3 EXPT(S)

DICv1.0 integrates attention and expectation modules into existing captioning architectures, estimating causal expectations for visual, semantic, and vocabulary representations before producing word distributions. It is trained with an interventional target followed by reinforcement learning.

  • EXPT modules: EXPT estimates conditional expectations by combining learned representation dictionaries with query-key operations.The dictionary approximates the representation space, while the context vector acts as the query and dictionary elements act as keys.
  • Decoder architecture: The decoder feeds attended visual features and recurrent states into EXPT modules and the embedding layer to compute word distributions.The estimated vectors correspond to E[z|X,h][z], E[s|h][s], and E[x|h][x].
  • Training: The models use P(L∗|do(X)) as the cross-entropy target for the first 35 epochs, then optimize a reinforcement-learning loss for 65 epochs.The reinforcement-learning reward can be a sentence-level metric such as CIDEr-D.

5 EXPERIMENTS

Experiments on MS COCO evaluate deconfounding through ablations, bias analyses, human judgments, network-size controls, and comparisons with state-of-the-art captioners. DICv1.0 improves caption similarity and reduces measured bias, while its evaluation also exposes difficult bias cases and metric limitations.

  • Experimental Setup: MS COCO experiments use Karpathy and official online splits, with CIDEr-D, BLEU, METEOR, ROUGE, SPICE, CHAIR, and word-accuracy metrics.The study also includes human evaluations of caption consistency and attention quality.
  • Deconfounding Techniques: UD-DICv1.0 raises CIDEr-D from 125.3 to 128.8 over UD, while larger backdoor dictionaries improve similarity scores up to 127.5 for UD-BD/CC.These ablations assess the contribution of deconfounding techniques and dictionary size.
  • Deconfounding Techniques: Sequential comparisons show that adding deconfounding for D and S improves performance, and deconfounding S outperforms merely adding more dictionary parameters.The authors conclude that mediators or ConceptNet alone are insufficient without identifying and deconfounding hidden confounders.
  • Bias Analysis: UD-DICv1.0 achieves the lowest CHs&CHi and highest accuracy for gender, attribute, action, and quantifier words among the compared ablations.The reported bias analysis evaluates word accuracy when specific visual concepts appear.
  • Bias Analysis: For IDBed, gender-word accuracy rises from below 0.6 with UD to almost 0.9 with UD-DICv1.0, whereas IDHorse action bias remains below 0.4 across three models.The authors attribute persistent action-bias difficulty to language and selection biases in human annotations.
  • Network Size and State of the Art: Adding LSTM layers alone improves CIDEr-D but does not clearly reduce CHs or CHi, whereas UD-DICv1.0 reaches CIDEr-D 128.8 with stronger bias reduction.State-of-the-art comparisons report CIDEr-D scores of 128.8 for UD-DICv1.0 and 131.1 for AoA-DICv1.0, with the two models achieving the highest online-test c5 and c40 scores.

6 CONCLUSIONS

The paper concludes that DICv1.0 emerges from a causal retrospect of image captioning and offers a starting point for further development, with evaluation reported on the online MS COCO test server.

  • The causal retrospect motivates DICv1.0 as a framework for addressing dataset bias in modern captioners.
  • DICv1.0 is derived from the paper’s causal retrospect and presented as a starting point for future image-captioning research.
  • Table 6 reports single-method performances on the online MS COCO test server using CIDEr-D scores.The table separates results trained with CIDEr-D computed from 5 captions from those trained on the whole training set.

Supplementary Material of Deconfounded Image

The supplementary material introduces causal-graph preliminaries for deconfounding image captioning, including backdoor paths, junction types, and rules for blocking information flow.

  • Causal preliminaries: A causal graph represents causal or anticausal links with directed arrows, while information can pass in either direction.The supplement uses X → L as a causal link and X ← D as an anticausal link.
  • Causal preliminaries: A backdoor path from X to L starts with an arrow entering X and can create a non-causal route between variables.Examples include X ← D → L and paths involving intermediate variables.
  • Causal preliminaries: Deconfounding X and L requires blocking every backdoor path so the causal effect P(L|do(X)) can be calculated.For the example graph, blocking X ← D → L is sufficient.
  • Causal preliminaries: The supplement distinguishes chain, confounding, and collider junctions, each with different information-flow behavior.The chain and confounding junctions transmit information through the middle variable, whereas the collider behaves oppositely.
  • Causal preliminaries: Controlling the middle variable blocks information flow in chain and confounding junctions but must be avoided for a collider.

B.1 Derivations of Eq. (17) and Eq. (21)

The derivation computes sentence- and word-level interventional caption distributions by combining partial effects and controlling variables that block relevant backdoor paths.

  • Sentence-level derivation: The sentence-level P(L|do(X)) is derived through a front-door procedure that computes P(Z|do(X)) and P(L|do(Z)) before chaining them.This derivation corresponds to Eq. (17) of the manuscript.
  • Sentence-level derivation: The backdoor paths from X to Z are naturally blocked because they contain collider structures involving D, L, and Z or S.
  • Sentence-level derivation: Computing P(L|do(Z)) requires controlling S and X to block the two backdoor paths Z ← S → L and Z ← X ← D → L.
  • Word-level derivation: At the word level, the interventional distribution is conditioned on H=h, where H denotes accumulated context from the partially generated caption.The value of H is pre-computed at each generation step.
  • Word-level derivation: The derivation treats paths through H as blocked because H is already controlled to h.The supplement notes that H appears only in confounding junctions.
  • Word-level derivation: The resulting word-level expression is identified as Eq. (21), while conditioning S on h is retained experimentally to increase representation power.Without this conditioning, the expected S representation would degrade to a fixed vector.

B.2 Derivations of Eq. (19) and Eq. (20)

The supplement derives the network approximations by replacing expectations with normalized weighted geometric means and using Softmax normalization to obtain valid caption distributions.

  • NWGM approximation: NWGM approximates the expectation of a function in deep networks whose final layer is Softmax.The approximation is used to derive Eq. (19).
  • NWGM approximation: The weighted geometric mean places distribution weights inside exponential terms, enabling expectations to be absorbed into the exponential representation.
  • Model application: The model parameterizes p(L|s, x, z) with a network ending in a Softmax layer before applying the approximation.
  • Model application: The resulting approximation combines expectations over s, x, and z in an exponential expression for the caption distribution.
  • Model application: Because p(L|s, x, z) is only proportional to exp[g(s, x, z)], the derivation uses approximation rather than equality and applies Softmax normalization.Normalization ensures that P(L|do(X)) sums to 1.
  • Word-level approximation: At word level, NWGM is used again to approximate the distribution in Eq. (F), with EXPT modules estimating the required expectations.Conditioning S on h is retained because otherwise E[s] would remain constant during caption generation.

C NETWORK ARCHITECTURE

The two deconfounded captioners use attention modules within a shared decoder structure, with implementation details specified for Top-Down Attention and Attention on Attention.

  • Module specifications: The appendix provides detailed structures for the EXPT module, Top-Down Attention, and Attention on Attention.The supplied materials identify these details through Tables A, B, and C.
  • ATT modules: UD-DICv1.0 and AoA-DICv1.0 deploy Top-Down Attention and Attention on Attention, respectively.These are the two ATT modules used in the models.
  • Attention on Attention: Attention on Attention computes multihead self-attention by concatenating eight heads and applying a trainable projection matrix.The listed formulation produces M = Cat(head1, ..., head8)W_C before the output transformation.
  • Bias context: The architecture section also includes a figure describing three different types of dataset biases.The supplied figure is labeled Fig. C.
  • Decoder structure: The common decoder structure uses either an LSTM or GLU for g(·), depending on the captioner.The ATT module is likewise selected according to the model: Top-Down attention for UD-DICv1.0 and Attention on Attention for AoA-DICv1.0.

D.1 Effects of the Dataset Biases.

The section examines several dataset-bias patterns and reports that deconfounding improves attention and robustness, although balancing one bias does not eliminate all gender-related effects.

  • Dataset bias types: Exhaustively quantifying dataset bias is nearly impossible because the confounder composition is complex.The section therefore analyzes selected bias types rather than enumerating every possible combination.
  • Dataset bias types: Dataset biases include long-tail word and concept frequencies, gender-object associations, and co-occurrence patterns among multiple words.Examples include frequent “person” descriptions, gender-linked concepts, and “ride” being favored after “person” and “horse.”
  • Balanced-data experiment: 34,576 captions are deleted to balance gender-object co-occurrences before training and testing the compared models.The remaining captions are used for UD-DICv1.0 and UD.
  • Balanced-data experiment: UD-DICv1.0* has lower CIDEr-D loss with fewer data, while its advantage over UD* weakens on the more balanced dataset.The authors attribute the reduced-data benefit partly to additional ConceptNet knowledge and report stronger A@Gen improvement for UD* under balancing.
  • Balanced-data experiment: Balancing gender and single-concept co-occurrences improves accuracy but does not reach 100%, indicating remaining sources of gender bias.The passage points to nonvisual concepts and combinations of concepts as possible additional factors.
  • Attention comparison: UD-DICv1.0 can attend to the person rather than the computer and generate “woman” correctly in the illustrated gender example.The baseline instead focuses on the computer and exploits its co-occurrence with “male.”

E.2 Experiments and Results

Experiments compare deconfounded captioners with baselines and report stronger CIDEr-D performance, while attention comparisons illustrate a shift toward relevant visual regions.

  • VATEX experiments: The VATEX experiment compares UD-DICv1.0 and AoA-DICv1.0 with captioners using the same visual features and LSTM-based decoder.The comparison includes UP- DOWN, VATEX, AoA, HRN, and ORG.
  • VATEX experiments: The evaluated models use ConceptNet commonsense structures and cross-entropy training for the VATEX comparison.The implementation searches nouns, verbs, and adjectives appearing more than 20 times in the training set.
  • VATEX experiments: UD-DICv1.0 and AoA-DICv1.0 achieve higher CIDEr-D scores than the compared VATEX models.The authors interpret these comparisons as confirming an advantage for deconfounding techniques in video captioning.
  • Attention comparison: The attention comparison contrasts the attended regions of UD-DICv1.0 with those of UD.The figure presents the deconfounded model above and the baseline below in each case.
Loading 2003.03923v2…