PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Causal Feature Learning

Krzysztof Chalupka, Pietro Perona, Frederick Eberhardt

arXiv:1412.2309v2stat.MLcs.AIcs.CVcs.LG

TL;DR

The paper addresses how to identify visual features that cause behavior when causal variables must be constructed from raw pixels and may be confounded by noncausal correlates. It defines visual causes as causal macro-variables, proves the Causal Coarsening Theorem, and learns manipulator functions for targeted image interventions. Experiments on synthetic and real data illustrate the inference and manipulation algorithms, while the manipulator does not verify that other system variables remain unchanged.

  • Problem

    Identifying a visual cause requires distinguishing causal image features from correlates, even when noncausal features predict the target behavior equally well or more strongly.

  • Method

    The paper constructs causal macro-variables from pixels, uses the Causal Coarsening Theorem for observational inference with limited experiments, and learns manipulator functions for image interventions.

  • Results

    The inference and learning algorithms are illustrated in experiments using synthetic and real data; manipulator error decreases from almost 40% on Iteration 1 to about 6% on Iteration 20.

  • Takeaways & Limitations

    The framework provides a causal account of visual features that extends causal graphical models to settings where causal variables must be constructed from micro-variables.

  • Takeaways & Limitations

    The manipulator uses closest-image heuristics and does not check whether other variables, particularly spurious correlates, remain unchanged.

Abstract

from arXiv · show

We provide a rigorous definition of the visual cause of a behavior that is broadly applicable to the visually driven behavior in humans, animals, neurons, robots and other perceiving systems. Our framework generalizes standard accounts of causal learning to settings in which the causal variables need to be constructed from micro-variables. We prove the Causal Coarsening Theorem, which allows us to gain causal knowledge from observational data with minimal experimental effort. The theorem provides a connection to standard inference techniques in machine learning that identify features of an image that correlate with, but may not cause, the target behavior. Finally, we propose an active learning scheme to learn a manipulator function that performs optimal manipulations on the image to automatically identify the visual cause of a target behavior. We illustrate our inference and learning algorithms in experiments based on both synthetic and real data.

1 INTRODUCTION

The paper develops a causal framework for identifying visual features that cause behavior rather than merely correlate with it. It defines visual causes as causal macro-variables constructed from pixels, connects them to graphical models and machine-learning inference, and proposes manipulations for learning them.

  • Visual causal variables: Visual causes are macro-variables constructed from image pixels that contain all image-available causal information about a target behavior.The framework places these variables within causal graphical models.
  • Inference from observation: The Causal Coarsening Theorem shows how observational data can learn visual causes with minimal experimental effort and connects the result to standard classification tasks.The theorem supports estimating causal behavior classes from representative images within observational classes.
  • Active manipulation: The paper proposes learning a manipulator function that automatically performs perceptually optimal image manipulations on visual causes.The framework is presented as applicable beyond vision to other aggregates of micro-variables when manipulation is possible.
  • Constructing causal variables: The approach extends causal feature learning to settings where candidate causal variables must be constructed from raw micro-variables before causal relations can be established.This differs from standard causal graphical-model settings, where macro-variables are already specified.
  • Confounding and intervention: The framework explicitly separates observational association from interventional causal effect in the presence of possible confounding.It distinguishes P(T|I) from P(T|do(I)).
  • Causal versus spurious features: The framework distinguishes causal visual features from spurious correlates that may predict behavior equally well or more strongly without affecting it under manipulation.In the example, h-bars cause the target behavior, whereas v-bars correlate through a hidden common cause.

2 A THEORY OF VISUAL CAUSAL FEATURES

The paper defines visual causes as causal macro-variables constructed from image pixels, distinguishing them from observationally predictive but noncausal features. It formalizes this distinction with causal and observational partitions and shows how causal and spurious information can be compressed into macro-variables.

  • 2.1 VISUAL CAUSES AS MACRO-VARIABLES: A visual cause is a high-level feature of an image that contains all causal information about a target behavior available in the pixels.The feature is constructed from constitutive image variables rather than treated as caused by them.
  • 2.2 GENERATIVE MODELS: FROM MICRO- TO MACRO-VARIABLES: The generative model represents images as produced by hidden variables, while behavior depends on the image and possibly hidden confounders.Independent noise affecting behavior is marginalized from the model.
  • 2.2 GENERATIVE MODELS: FROM MICRO- TO MACRO-VARIABLES: The observational partition groups images by P(T | I), whereas the causal partition groups them by intervention-based behavior probabilities.These partitions distinguish predictive similarity from equality under image manipulation.
  • 2.2 GENERATIVE MODELS: FROM MICRO- TO MACRO-VARIABLES: The framework assumes binary behavior and discrete image and hidden variables, while continuous behavior is outside the article’s scope.The authors state that extending to non-binary discrete behavior is straightforward but notation becomes more complicated.
  • 2.3 THE CAUSAL COARSENING THEOREM: The Causal Coarsening Theorem states that, for almost all compatible generative distributions, the causal partition coarsens the observational partition.Thus, observational classes may subdivide causal classes, while the measure-zero exceptions can be nonempty.
  • 2.3 THE CAUSAL COARSENING THEOREM: Observationally informative features can reflect other non-visual causes rather than cause behavior, but the theorem permits causal labels to remain constant within observational classes.This supports causal inference from observational classifications with limited experimentation.
  • 2.4 VISUAL CAUSES IN A CAUSAL MODEL CONSISTING OF MACRO-VARIABLES: The visual cause C and spurious correlate S together preserve all information about T in the image, while only C carries causal information.Theorem 7 states P(T | I) = P(T | C, S), and no equally informative macro-variable has lower Shannon entropy than H(C, S).
  • 2.4 VISUAL CAUSES IN A CAUSAL MODEL CONSISTING OF MACRO-VARIABLES: In some cases, manipulating C to a desired value is impossible without also changing S.This is presented as a physical constraint on interventions in the image space.

3 CAUSAL FEATURE LEARNING: INFERENCE ALGORITHMS

The paper learns visual causes from observational and causal data, then uses them to construct image manipulators that achieve desired causal effects while minimizing image distance. Its inference procedure exploits causal coarsening, and its active manipulator-learning procedure iteratively improves causal accuracy.

  • Problem formulation: The framework learns a causal variable C and a manipulator MC that returns a maximally similar image with a desired causal effect.The manipulator is defined as the closest image belonging to the target causal class under metric d.
  • Manipulator function: The manipulator searches over images with the desired causal effect, but preserving spurious correlates is only a heuristic rather than a checked guarantee.The selected image is closest under d, while other system variables may change.
  • Causal inference: Observational predictors can exploit spurious information, so low test error alone does not establish that learned image features are causal.Causal feature learning distinguishes predictive associations from features that change the target under intervention.
  • Causal inference: The Causal Coarsening Theorem permits estimating one representative per observational class to infer causal labels for the entire class, ignoring sampling issues.The causal partition coarsens the observational partition, so members of one observational class share the relevant interventional distribution.
  • Manipulator learning: Algorithm 2 trains a causal neural network, chooses random manipulation starting points and target causal classes, queries an oracle, and iteratively updates the manipulator.Each iteration evaluates Q manipulations using manipulation error and average manipulation distance.
  • Manipulator learning: The procedure evaluates both causal accuracy through manipulation error and minimality through average manipulation distance.The algorithm can stop when one or both criteria reach desired values.

4 EXPERIMENTS

Experiments on GRATING and MNIST demonstrate that the proposed algorithms can learn causal visual features and train manipulators using synthetic and real data. GRATING isolates a causal feature from a correlated distractor, while MNIST shows progressive class-directed image changes through human-annotator training.

  • Experimental setup: The experiments use synthetic GRATING data and handwritten-digit MNIST data to test causal feature learning and manipulator training.The authors emphasize feasible algorithmic solutions rather than advanced computer vision performance.
  • GRATING: In GRATING, observational images contain one vertical or horizontal bar with approximately 3% random-noise pixels, and Algorithms 1 and 2 learn the visual cause.The restrictions simplify the binary-image experiment and improve presentation clarity.
  • GRATING: After twenty manipulator iterations, 6 of 100 GRATING manipulations are unsuccessful, while irrelevant pixels are better preserved than at iteration 1.The fully trained manipulator changes the h-bar, which causes T, while ignoring the correlated but noncausal v-bar.
  • GRATING: GRATING manipulation error decreases quickly over ten iterations while manipulation distance remains nearly constant.The figure contrasts training progress with original and manipulated GRATING images.
  • MNIST: For MNIST, human annotators train manipulators separately for all ten digits, and manipulation error decreases as training proceeds.The visualizations use randomly selected digits rather than cherry-picked examples.
  • MNIST: Successive MNIST manipulators progressively remove original-digit features and add features of the target class.The task modifies images so they become or cease resembling a target digit such as 7.

5 DISCUSSION

The discussion connects causal feature learning to neural networks and causal discovery. The framework constructs well-defined causal macro-variables from raw micro-variables, enabling causal analysis where candidate causes are not already specified.

  • Causal reasoning and neural networks: The manipulator-learning procedure uses causal reasoning to improve standard correlational classifier boundaries and aims to extract truly causal features.The paper presents this as a connection between causal reasoning and neural network models, including robustness against adversarial examples.
  • Causal discovery: The framework addresses causal-discovery settings in which causal variables are aggregates of other variables by specifying how macro-variables supervene on micro-variables.This responds to the problem that standard causal discovery often presupposes meaningful, well-defined causal variables.
  • Scope: Constructing well-defined causal macro-variables is presented as essential for causal methodology in domains lacking clearly delineated candidate causes.The discussion names vision, neuroscience, climate science, and economics as examples of such domains.

6 APPENDIX: PROOF OF THE CAUSAL COARSENING THEOREM

The appendix proves the Causal Coarsening Theorem by expressing violations of causal coarsening as nontrivial polynomial constraints and showing that their solution set has measure zero. The proof first establishes an auxiliary theorem, then extends the argument to fixed observational partitions.

  • Auxiliary theorem: The Auxiliary Theorem states that models whose causal partition is not a coarsening of the observational partition form a Lebesgue measure-zero subset.This establishes the generic relationship before the proof handles fixed observational partitions.
  • Parameterization: The proof parameterizes joint distributions using α, β, and γ, fixing α and γ while varying β over a lower-dimensional simplex.This isolates the parameters needed to analyze the causal coarsening constraint.
  • Constraint: Violations occur when two images have equal observational target distributions but different causal classes, which becomes a polynomial constraint on the parameter space.The constraint is written for image pairs i and j with P(T|i) = P(T|j) but C(i) ≠ C(j).
  • Measure-zero argument: A nontrivial polynomial constraint holds on a measure-zero set, and finitely many image pairs preserve this measure-zero conclusion for all violations.The proof then applies Fubini’s theorem to the full parameter space.
  • Main theorem: The Causal Coarsening Theorem concludes that, among models inducing a fixed observational partition, almost all causal partitions are coarsenings of that partition.The final restricted-parameter-space argument follows the Auxiliary Theorem.

7 APPENDIX: CCT EXAMPLES AND COUNTER-EXAMPLES

The appendix presents three distributions illustrating the possible relationships between observational and causal partitions. Only the first, in which the causal partition coarsens the observational partition, is the generic CCT-consistent case.

  • Examples: The first model has a causal partition that properly coarsens the observational partition, agreeing with the Causal Coarsening Theorem.The causal partition merges observational classes without violating the theorem.
  • Examples: The second model has an observational partition that properly coarsens the causal partition, a measure-zero case requiring carefully aligned parameters.This relationship is possible but nongeneric under the theorem.
  • Examples: The third model has incompatible causal and observational partitions, with neither partition coarsening the other; this is also a measure-zero case.The appendix provides models and software for verifying these partition computations.

8 APPENDIX: PROOF OF THE COMPLETE MACRO-VARIABLE DESCRIPTION THEOREM

The Complete Macro-variable Description Theorem shows that the pair of macro-variables C and S preserves the target distribution available from the image while using no more Shannon entropy than any equivalent representation.

  • Completeness: The pair C and S satisfies P(T | I) = P(T | C, S), so it preserves the target-behavior distribution conditioned on the image.This is the theorem’s completeness statement for the macro-variable description.
  • Minimality: Any variable X satisfying P(T | I) = P(T | X) has Shannon entropy H(X) ≥ H(C, S).Thus C and S provide an information-preserving representation with minimal entropy among equivalent variables.

9 APPENDIX: PREDICTIVE NON-CAUSAL INFORMATION IN CAUSAL VARIABLE C

The appendix shows that the causal variable C can retain predictive information that is not causal when hidden confounding is present. In the illustrated binary case, observational and causal partitions coincide even though P(T|I) differs from P(T|do(I)).

  • Predictive non-causal information: With the graph I → T and I ← H → T, the visual cause C can retain predictive information that is not causal.The hidden variable H confounds the image and target behavior.
  • Observational versus causal: The general case satisfies P(T|I) ≠ P(T|do(I)), distinguishing observational prediction from intervention-based causal effect.This difference motivates separating predictive information from causal information.
  • Binary example: In the binary example, the observational partition equals the causal partition, so the spurious-correlate variable S is constant and can be omitted.No further distinctions remain within the causal classes.

10 APPENDIX:THE MNIST ON MTURK EXPERIMENT

The MNIST experiment trained one-vs-all neural networks to manipulate digit images toward opposing classes, then used human annotations to iteratively refine the manipulators.

  • Manipulator training: Ten one-vs-all neural networks were trained with cross-validation over several architectures, maxout activations, stochastic gradient descent, and dropout.The candidate architectures included one- and two-layer networks with 100 or 300 hidden units.
  • Image manipulation: Each network transformed 1000 training images into maximally similar images of the opposing class.
  • Manipulated datasets: The experiment created ten manipulated datasets, each targeting one digit class against non-class or other-digit images.For example, the zero dataset included zeros manipulated to be non-zeros and other digits manipulated to be zeros.
  • Human annotation: Amazon Mechanical Turk annotators labeled the randomly mixed manipulated images, with each task containing 250 images.Annotators entered digit labels or “?” through the interface shown in Figure 8.
  • Iterative refinement: The annotations were added to the corresponding datasets before training the next iteration of manipulator networks.This update-and-retrain process continued until manipulator training was complete.
Loading 1412.2309v2…