Deconfounded Visual Grounding

Introduction

• Visual grounding, also called referring expression comprehension, extends detection from fixed noun vocabularies to free-form natural language queries.The paper focuses on one-stage methods, while noting that the two-stage and one-stage pipelines differ little when gradients pass through the detector.Source: Introduction · p. 1
• Language patterns can bias predictions toward typical locations or box sizes, even when the region does not satisfy the query.Sheep queries tend to predict central regions, while “corner of” queries tend toward smaller regions.Source: Introduction · pp. 1–2
• The authors frame the visual grounding pipeline as a causal graph to explain how language, vision, locations, and an unobserved confounder interact.The confounder is generally unobserved and non-enumerative in this multimodal task.Source: Introduction · p. 2
• RED estimates a substitute confounder from observational data, requiring no prior knowledge of the confounder.The method is implemented as language attention whose embedding replaces the language embedding in a grounding method.Source: Introduction · p. 2

Related Work

• Visual grounding methods have used both two-stage detection-and-ranking pipelines and end-to-end one-stage pipelines that fuse image and language features.The paper deploys multiple one-stage methods as baselines to demonstrate RED’s generic applicability.Source: Related Work · p. 2
• The paper applies causal deconfounding to visual grounding and generates a proxy for the unobserved confounder, drawing on the Deconfounder approach (Wang and Blei 2019).Figure 3 contrasts conventional duplicated language features with RED’s deconfounded language features and dictionary extraction.Source: Related Work · p. 3, Figure 3 · p. 3
• RED builds a fixed confounder dictionary before grounding, unlike approaches that enumerate confounder values or learn the dictionary jointly.The authors state that this setup guarantees the confounder set causes the feature, not vice versa.Source: Related Work · p. 3

Causal Graph

• The graph represents image X, query R, location L, and unobserved confounder G, with G directly causing L and linking X and R.G is dataset-specific; the graph’s G → L link captures how dataset context shapes annotated locations.Source: Causal Graph · p. 3, Visual Grounding in Causal Graphs · p. 3, Figure 4 · p. 4

Causal Inference Process

• Even conditioning on image X leaves the backdoor path R ← G → L, so observational and intervention probabilities for location can differ.The target is a location prediction that varies with the query; the paper distinguishes this language-related confounding from the broader confounding associated with intervening on X.Source: Causal Inference Process · pp. 3–4

Deconfounder

• Because G is unobserved, RED estimates a substitute confounder from a query-generation model and averages over its strata to approximate the intervention probability.The model is intended to capture confounder strata by making query features independent conditional on the substitute confounder; the approach uses a clustered dictionary to make adjustment more efficient.Source: Deconfounder · p. 4, Referring Expression Deconfounder (RED) · p. 4

Referring Expression Deconfounder (RED)

• RED’s generative-model step uses an auto-encoder to encode query R as substitute confounder Ĝ and decode it back to query R′.The formulation also supports nondeterministic generative models, including VAEs and conditional GANs.Source: Referring Expression Deconfounder (RED) · p. 4
• RED implements feature fusion as language attention and replaces the original language embedding with deconfounded feature r′ in the grounding model.The method fuses r and Ĝ without image feature X, as the intervention is on R; the passage reports that other fusions taking X as input hurt performance.Source: Referring Expression Deconfounder (RED) · p. 5

Datasets

• Evaluation uses RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities, with RefCOCO and RefCOCO+ including testA and testB splits.The RefCOCOg partition is RefCOCOg-umd, reported as val-u and test-u.Source: Datasets · p. 5

Implementation Details

• Implementation builds a 10-cluster confounder dictionary from language embeddings encoded by frozen uncased BERT, then uses RED’s deconfounded embeddings in grounding training.The same frozen BERT structure is used for dictionary construction and grounding embeddings to maintain consistency.Source: Implementation Details · p. 5

Quantitative Results

• RED generally improves SOTA grounding methods; gains are larger on ReferItGame than Flickr30K Entities, with increases of 3.16% vs. 0.25%.The reported comparisons cover RefCOCO-series, ReferItGame, and Flickr30K Entities.Source: Quantitative Results · p. 6, Table 1 · p. 6, Table 2 · p. 6
• On RefCOCO and RefCOCO+, RED improves more on testB than on testA or val, while results vary with the baseline and deconfounding configuration.The paper attributes the testB pattern to its distribution being more discrepant from training; it reports that clustering counts from 5 to 20 improve over baseline, with no further gains above 10, and Euclidean distance consistently superior. The vision-and-language deconfounder and attention on both image and language features perform worse than RED’s language-focused approach.Source: Quantitative Results · pp. 6–7, Table 3 · p. 6
• Yang’s-V1 runs at 24.56 ms per sample and Yang’s-V1+RED at 26.53 ms, an 8% inference overhead on a Tesla V100.The authors describe this overhead as reasonable.Source: Quantitative Results · p. 7

Qualitative Results

• Qualitative examples show RED correcting location-biased predictions in two cases, while another example remains biased toward a colorful computer screen over a salient bench.The figure also presents failure cases.Source: Qualitative Results · p. 7, Figure 6 · p. 7

• RED seeks a substitute confounder from observational data and implements it as language attention that replaces language embeddings in grounding methods.The authors report consistent improvements to various strong baselines.Source: Conclusions · p. 7
• The authors identify future work in trying other generative models and adding large-scale vision-language pretraining priors to the deconfounder.They motivate the latter direction by noting that addressing an unobserved confounder is generally ill-posed.Source: Conclusions · p. 7

Causal Graph

• A causal graph represents variables as nodes and causal relationships as directed arrows, with chain and fork structures among its basic forms.The section uses the fork to illustrate how a common cause can create a non-causal association between two variables.Source: Causal Graph · p. 10

Confounder

• A confounder can open a backdoor path between variables, making them spuriously correlated even when one does not cause the other.The ice-cream-sales and crime example uses temperature as a common cause; observed dependence does not establish a causal effect.Source: Confounder · p. 10

Do and Backdoor Adjustment

• Backdoor adjustment uses variables that block all backdoor paths to estimate causal effects from observational data, but unobserved confounders cannot be controlled directly.The section distinguishes intervention, denoted do(x), from observing X=x and describes adjustment as a way to express the intervention using probabilities.Source: Do and Backdoor Adjustment · pp. 10–11

Ignorability and Deconfounder

• When a confounder is unobserved, a substitute that satisfies ignorability can support backdoor adjustment under the stated assumptions.The described Deconfounder approximates a multi-cause confounder and relies on the absence of single-cause confounders.Source: Ignorability and Deconfounder · p. 11

Detailed Proof for Deconfounder

• The proof argues that independence among causes conditional on the substitute confounder rules out an unobserved multi-cause confounder, under the assumption that no single-cause confounder exists.This supports using the substitute confounder for backdoor adjustment.Source: Detailed Proof for Deconfounder · p. 12

Detailed Proof for Equation 5 in Deconfounder

• The Equation 5 derivation uses normalized weighted geometric mean to move an expectation to the feature level and avoid massive calculation.The final approximation extracts X because it is not involved in the logit expectation over the substitute confounder.Source: Detailed Proof for Equation 5 in… · p. 12

Implementation Details for Deconfounder

• The implementation details describe an auto-encoder and a top-down attention operation, with language features projected to the substitute-confounder dimension.Table 1 is identified as giving the auto-encoder implementation details.Source: Implementation Details for Deconfounder · p. 12, Table 1 · p. 12

More Details for Dataset and Training Settings

• The supplementary settings describe dataset characteristics, visual representations, and optimization, while Table 2 covers auto-encoder and fusion-method ablations.RefCOCO+ avoids absolute location words; the described experiments use a Darknet53 backbone and a unified batch size of 64.Source: More Details for Dataset and Training… · pp. 12–13, Table 2 · p. 13

• Table 2 provides additional ablation examples on RefCOCO, RefCOCO+, and RefCOCOg.The supplied passage identifies the table’s scope but does not report its cell values or comparisons.Source: More Quantitative Examples · p. 13

More Qualitative Examples

• Additional qualitative examples support the language-bias corrections made by RED.Source: More Qualitative Examples · p. 13