PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Commonsense R-CNN

Tan Wang, Jianqiang Huang, Hanwang Zhang, Qianru Sun

arXiv:2002.12204v3cs.CV

TL;DR

Visual systems struggle with commonsense reasons and relationships because visual observations and language corpora can encode biased or incomplete context. VC R-CNN learns reusable region features with causal intervention, and experiments show significant, consistent improvements across high-level vision tasks while the method remains lightweight; it does not claim a philosophically complete definition of visual common sense.

  • Problem

    Visual systems remain weak at high-level commonsense reasons, while image co-occurrences and multimodal text can reflect confounding or reporting bias.

  • Method

    VC R-CNN learns unsupervised region representations by predicting contextual objects with causal intervention P(Y |do(X)) and provides features that can be concatenated into task networks.

  • Results

    VC R-CNN produces significant and consistent improvements over strong baselines across Image Captioning, VQA, and VCR, while remaining as fast and memory-efficient as Faster R-CNN.

  • Takeaways & Limitations

    Causal intervention offers a practical direction for learning visual commonsense features that transfer across high-level vision tasks without rewriting task networks.

  • Takeaways & Limitations

    VC R-CNN does not provide a philosophically correct definition of visual common sense, only a computational definition based on observed objects and hidden causalities.

Abstract

from arXiv · show

We present a novel unsupervised feature representation learning method, Visual Commonsense Region-based Convolutional Neural Network (VC R-CNN), to serve as an improved visual region encoder for high-level tasks such as captioning and VQA. Given a set of detected object regions in an image (e.g., using Faster R-CNN), like any other unsupervised feature learning methods (e.g., word2vec), the proxy training objective of VC R-CNN is to predict the contextual objects of a region. However, they are fundamentally different: the prediction of VC R-CNN is by using causal intervention: P(Y|do(X)), while others are by using the conventional likelihood: P(Y|X). This is also the core reason why VC R-CNN can learn "sense-making" knowledge like chair can be sat -- while not just "common" co-occurrences such as chair is likely to exist if table is observed. We extensively apply VC R-CNN features in prevailing models of three popular tasks: Image Captioning, VQA, and VCR, and observe consistent performance boosts across them, achieving many new state-of-the-arts. Code and feature are available at https://github.com/Wangt-CN/VC-R-CNN.

1. Introduction

VC R-CNN addresses visual systems’ difficulty with high-level commonsense reasoning by learning region features through causal intervention rather than conventional likelihood. The resulting extractor improves downstream vision tasks while remaining lightweight and non-intrusive, although it does not claim a philosophically complete definition of visual common sense.

  • Motivation: Visual systems often recognize what and where but struggle with high-level commonsense reasons and relationships.Dataset bias can produce incorrect visual relationships or unreasonable visual attention in captioning and VQA.
  • Motivation: Language-based contextual prediction can learn useful word vectors, but image co-occurrences may confound commonsense relations with spurious context.For example, keyboard and mouse may be wrongly associated with tables instead of their underlying computer relationship.
  • Method: Causal intervention P(Y |do(X)) borrows and inserts objects from other images to test whether X causes Y despite contextual confounders.This intervention replaces context-dependent probabilities with a prior over borrowed objects, alleviating observational bias.
  • Method: VC R-CNN combines an R-CNN visual backbone with causal intervention as an unsupervised training objective for reusable region features.Its learned features target high-level tasks including Image Captioning, VQA, and VCR.
  • Contribution: VC R-CNN yields significant and consistent improvements over strong task-specific baselines while matching Faster R-CNN in speed and memory efficiency.Downstream networks need only feature concatenation rather than architectural rewriting.
  • Scope: The paper presents a computational approximation of visual common sense, not a philosophically correct definition.Its definition combines unsupervised learning from observed objects with pursuit of causalities hidden in those observations.

2. Related Work

Related work learns visual-semantic features from multimodal data, images, knowledge bases, or videos, but faces reporting bias, annotation challenges, or downstream cost. VC R-CNN instead learns from images alone and supports simple feature concatenation.

  • Multimodal Feature Learning: Weakly supervised multimodal feature learning can encode visual-semantic knowledge but is affected by language reporting bias and costly downstream fine-tuning.Common-sense relations may be omitted from text even when they are physically evident.
  • Visual Common Sense: VC R-CNN learns unsupervised visual features from images and concatenates them with original representations.This avoids relying on language text to record every commonsense relation.
  • Visual Common Sense: Prior visual common-sense methods learn from images with commonsense knowledge bases or learn actions from videos.The supplied related-work passage identifies these as two main categories of prior approaches.

3. Sense-making by Intervention

VC R-CNN replaces observational context prediction with causal intervention to reduce confounding and learn more sensible visual relationships. Its implementation approximates intervention over a practical confounder dictionary, with attention and collider filtering.

  • 3.1. Causal Intervention: Confounders can make toilet features encode surrounding sink-like context because P(z|toilet) overweights common restroom objects.The paper contrasts this biased likelihood with intervention, which adjusts the context weighting.
  • 3.1. Causal Intervention: Causal intervention P(Y |do(X)) cuts the confounder-to-X link, replacing P(z|X) with the prior P(z) so contexts contribute fairly.This is intended to alleviate observational context bias in visual feature learning.
  • 3.1. Causal Intervention: Intervention increases P(person|do(toilet)) relative to P(person|toilet) by reducing the influence of overrepresented restroom context.The cited example attributes the difference to the distributional gap between P(z|toilet) and P(z).
  • 3.1. Causal Intervention: VC R-CNN features separate window and leg clusters after intervention, while placing leg nearer head and window nearer wall in t-SNE visualizations.The method also produces multifaceted sub-clusters that vary across contexts.
  • 3.2. The Proposed Implementation: The practical proxy predicts contextual labels using a fixed N × d dictionary of category-averaged Faster R-CNN RoI features.The dictionary contains one averaged feature per category, such as 80 categories in MS-COCO.
  • 3.2. The Proposed Implementation: The implementation approximates the confounder expectation with attention-weighted dictionary features and removes possible colliders using NCC.NWGM is used to move the expectation into the Softmax efficiently, while attention ranks candidate confounders.

4. VC R-CNN

VC R-CNN uses a CNN backbone and paired RoI predictors to learn self and causal context representations. Its extracted features can be concatenated with original visual features for downstream models.

  • Architecture: VC R-CNN extracts object representations with RoIAlign and uses sibling Self Predictor and Context Predictor branches for each pair of RoI features.The Context Predictor estimates contextual labels with approximated do-calculus.
  • Training Objectives: The Self-Predictor loss classifies each RoI over N object categories using the ground-truth class.The Context Predictor instead predicts each context object with probabilities defined by P(Yi|do(X)).
  • Feature Extractor: VC R-CNN features are directly concatenated with original visual features for downstream tasks, without requiring task-network rewriting.The paper cautions against early concatenation in some self-attention models because it can slow training and affect computation.

5. Experiments

VC R-CNN was evaluated as an unsupervised visual feature across image captioning, VQA, and VCR, using concatenation with existing representations and intervention ablations. It produced broad gains, while performance depended on retaining conventional visual features and was constrained by question understanding in VQA.

  • Comparative Designs: Five feature settings compared original object features, pure VC features, detection-only features, correlation features, and the full intervention-based VC representation.The full +VC setting concatenates intervention-trained features with the original representation.
  • Results: Pure VC features hurt captioning performance, whereas concatenating VC with conventional visual features produced gains over detection and correlation baselines.The results indicate that object and attribute information remains useful alongside commonsense features.
  • Results: VC features achieved new state-of-the-art or superior performance across captioning, VQA, and VCR evaluations.Captioning results reported new state-of-the-art performance and best single-model scores; VQA achieved state-of-the-art results across three answer types; VCR remained superior across representations.
  • Results: VC features performed best on both standard captioning metrics and CHAIR, which measures object hallucination with lower scores indicating better performance.The reported advantage is attributed to intervention-based encoding of visual commonsense knowledge.
  • Transferability: Open Images training transferred effectively, with performances extremely close to MS-COCO training and reasonable improvements on the differently distributed VCR movie-clip data.The authors describe this as evidence of stability and semantic meaningfulness of the learned representation.
  • Ablation Study: Ablations found that random dictionaries and statistical E_z[z] reduced performance, while NCC refinement produced only a small downstream difference.The default fixed confounder dictionary with attention and NCC achieved the best results.

6. Conclusions

The paper presents VC R-CNN as an unsupervised visual feature representation based on causal intervention rather than conventional likelihood. Experiments report broad gains across strong baselines and metrics, with future work extending the approach to video and 3D point clouds.

  • Conclusions: VC R-CNN is an unsupervised feature representation method based on causal intervention and applicable to R-CNN frameworks.The extracted feature supports high-level tasks through feature concatenation.
  • Conclusions: Extensive benchmark experiments showed performance boosts on almost all strong baselines and metrics.The conclusion summarizes results across the paper’s evaluated tasks.
  • Conclusions: Future work will study VC R-CNN in modalities including video and 3D point clouds.

Supplementary Material

The supplementary material expands the paper’s causal-intervention derivation, implementation, architecture, experiments, and qualitative comparisons. It formalizes graph surgery and adjustment using invariant probabilities and intervention-induced independence.

  • Supplementary Scope: The supplement provides derivations and proofs for the do-expression and details the proposed implementation and network architecture.It also adds quantitative and qualitative experimental material.
  • Causal Intervention: Graph surgery removes confounder effects from Z to X, distinguishing pre-intervention probability P from manipulated probability Pm.The manipulated distribution preserves specified marginal and conditional properties.
  • Causal Intervention: Intervention makes Z and X independent, yielding the adjustment formula that averages conditional associations over the prior distribution P(z).This permits causal effects to be measured from conditional probabilities.
  • Examples: The person–toilet example shows intervention replacing P(Z|X) with P(Z), borrowing and placing contexts to alleviate long-tail distribution effects.

B. Our Proposed Implementation

The proposed implementation approximates intervention with a global confounder dictionary and normalized weighted geometric mean. Its formulation converts expectations over confounders into a practical feature transformation for context prediction.

  • Implementation: Figures compare P(z|Toilet) with P(z) and intervention-adjusted products across the full MS-COCO object-label set.The comparison illustrates how intervention substitutes prior context probabilities for context conditioned on X.
  • Implementation: The implementation represents confounders as a global dictionary Z=[z1,...,zN] with N dataset categories and feature dimension d.For MS-COCO, N is 80.
  • Implementation: Normalized weighted geometric mean approximates averaging softmax predictions over possible confounders in the context-label prediction task.The method changes a product formulation into a sum inside the exponential.
  • Implementation: Using the linear model fy(x,z)=W1x+W2·gy(z), the expected confounder feature E_z[gy(z)] becomes the input to the classifier.The RoI feature x remains unchanged while the confounder contribution is averaged.

B.2. Neural Causation Coefficient (NCC)

NCC is used to identify and handle collider structures that can distort causal reasoning in VC R-CNN. The implementation relies on synthetic causal data and qualitative causal-direction predictions, with strong collider cases discarded during training.

  • Collider structure: The collider is represented as a graph whose nodes are variables and whose arrows denote direct causal effects.Figure 4 illustrates the relevant causal graph structure.
  • NCC implementation: NCC models causal, anticausal, and non-causal relationships using synthetic observational samples.It trains on heteroscedastic additive-noise data generated from sampled mixture distributions and mechanisms.
  • NCC implementation: The NCC output qualitatively evaluates causal direction between real-world RoI feature vectors.At test time, the model can evaluate directly on RoI features.
  • Collider handling: VC R-CNN discards training samples with very strong collider structures using an NCC threshold of 0.001.This filtering addresses NCC’s possible large deviations on real-world features.
  • Limitations: NCC refinement produces only a small downstream-performance difference, possibly because VC R-CNN learns reasonable confounder attention during large-scale training.The authors identify further exploration of NCC and other causal-discovery methods as future work.

C. Network Architecture

The VC R-CNN architecture extracts paired RoI features, refines a category-based confounder dictionary with scaled dot-product attention, and combines effects from the input region and confounders.

  • Feature extraction: Paired RoI feature vectors x and y are extracted from an image using the feature-extraction backbone.The architecture specifies these extractions as the first two network components.
  • Confounder refinement: Scaled dot-product attention selects and refines confounders from the confounder dictionary Z.The dictionary is used as the source of confounder representations for the attention module.
  • Effect modeling: A linear addition model f_y(x, z) combines the effect on Y from X and confounder Z.This combination forms the final causal-effect modeling component described in the architecture.

D. More Quantitative Results

Additional experiments use bottom-up Faster R-CNN region features and show that concatenating VC features with vanilla region representations substantially improves image-captioning performance.

  • Base features: Bottom-up features provide salient object-level regions with attributes rather than uniformly sized image regions.They use pretrained Faster R-CNN with ImageNet and Visual Genome and enable attention over semantically meaningful regions.
  • Image captioning: Concatenating VC features with vanilla Faster R-CNN region representations leads to a huge image-captioning performance improvement.The comparison controls the number and locations of boxes using bottom-up-feature coordinates, with results reported in Table 2.

E.1. Failure Case

The paper documents failure cases in confounder refinement and downstream reasoning. Errors include implausible confounder attention, incorrect or incomplete downstream outputs, and failure to exploit the exact confounder needed to remove correlation bias.

  • Confounder refinement failures: VC R-CNN sometimes attends to inappropriate confounders when no obvious relation exists between X and Y.Examples include attending to giraffe for dog–vase intervention and skateboard for chair–fork intervention.
  • Confounder refinement failures: The authors propose choosing more appropriate context objects for the confounder dictionary as a future remedy.This is presented as a better schedule for confounder exploration.
  • Downstream task failures: In VQA, the model can produce a reasonable but incorrect answer, while image captioning may omit some instances.These examples are attributed partly to limited object detection and partly to failure to exploit the exact confounder.
  • VQA comparison: VC features improve VQA attention by reducing correlation bias compared with Faster R-CNN features.The reported example contrasts attention to a horse with more appropriate attention to a human when answering a question.
Loading 2002.12204v3…