Interventional Few-Shot Learning

Zhongqi Yue; Hanwang Zhang; Qianru Sun; Xian-Sheng Hua

Source-linked AI summary

Interventional Few-Shot Learning

Zhongqi Yue, Hanwang Zhang, Qianru Sun, Xian-Sheng Hua

arXiv:2009.13000v2Sep 28, 2020cs.LGcs.CV

TL;DR

The paper identifies pre-trained knowledge as a confounder that can hurt few-shot learning, especially for queries dissimilar to support samples. It proposes Interventional Few-Shot Learning using a structural causal model and backdoor adjustment, improving baselines across benchmarks and query hardness.

Problem
Pre-trained knowledge can act as a confounder in few-shot learning, degrading performance on queries dissimilar to support samples.
Method
Interventional Few-Shot Learning models the causalities among pre-trained knowledge, features, and labels, then applies backdoor adjustment through three practical implementations.
Results
IFSL improves all baselines across query hardness and achieves new 1-/5-shot state-of-the-arts on miniImageNet, tieredImageNet, and cross-domain CUB.
Takeaways & Limitations
IFSL is orthogonal to fine-tuning and meta-learning methods and provides a causal approximation to many-shot learning.
Takeaways & Limitations
Backdoor adjustment requires observing and stratifying the pre-trained knowledge, which is difficult for third-party pre-trained networks with unobserved datasets.

Abstract

from arXiv · show

We uncover an ever-overlooked deficiency in the prevailing Few-Shot Learning (FSL) methods: the pre-trained knowledge is indeed a confounder that limits the performance. This finding is rooted from our causal assumption: a Structural Causal Model (SCM) for the causalities among the pre-trained knowledge, sample features, and labels. Thanks to it, we propose a novel FSL paradigm: Interventional Few-Shot Learning (IFSL). Specifically, we develop three effective IFSL algorithmic implementations based on the backdoor adjustment, which is essentially a causal intervention towards the SCM of many-shot learning: the upper-bound of FSL in a causal view. It is worth noting that the contribution of IFSL is orthogonal to existing fine-tuning and meta-learning based FSL methods, hence IFSL can improve all of them, achieving a new 1-/5-shot state-of-the-art on \textit{mini}ImageNet, \textit{tiered}ImageNet, and cross-domain CUB. Code is released at https://github.com/yue-zhongqi/ifsl.

1 Introduction

The paper identifies pre-trained knowledge as a confounder in Few-Shot Learning and proposes Interventional Few-Shot Learning to counter its misleading effects. IFSL uses causal intervention and improves existing fine-tuning and meta-learning methods across several benchmarks.

Problem: Pre-trained knowledge can mislead Few-Shot Learning, especially when query samples are dissimilar to the support set.Stronger backbones improve average performance but can degrade performance on dissimilar queries, such as lion images associated with grass cues.
Proposed paradigm: Interventional Few-Shot Learning applies causal intervention to counter the deficiency introduced by pre-training.The proposed paradigm is motivated by the causal assumption connecting pre-trained knowledge, few-shot samples, and class labels.
Causal analysis: The paper models pre-trained knowledge, sample features, and labels with a Structural Causal Model in which pre-trained knowledge acts as a confounder.The model explains how spurious correlations can arise between support-set features and class labels.
Method: IFSL is orthogonal to downstream fine-tuning and meta-learning methods and can improve both types of baselines.The paper develops three implementations based on backdoor adjustment.
Results: IFSL achieves new 1-/5-shot state-of-the-art results on miniImageNet, tieredImageNet, and cross-domain CUB.Reported results are 73.51%/83.21% on miniImageNet, 83.07%/88.69% on tieredImageNet, and 50.71%/64.43% on cross-domain CUB.
Results: IFSL improves all baselines across similarities between support and query samples.The paper diagnoses performance across different S–Q similarities and reports consistent gains in every examined range.

2 Problem Formulations

Few-shot learning trains on small support sets, but pre-training can confound feature-label relationships, especially when query samples differ from support samples. The paper formulates this issue causally and proposes estimating P(Y|do(X)) through backdoor adjustment.

2.1 Few-Shot Learning: Few-shot learning trains a K-way classifier on an N-shot support set and evaluates it on a query set, typically with N=1 or 5 samples per class.
2.1 Few-Shot Learning: Fine-tuning uses a pre-trained network to extract representations and trains the classifier on the support set, while meta-learning additionally learns behavior from episodic tasks.
2.2 Structural Causal Model: The SCM models pre-trained knowledge D as causing feature representations X and the low-dimensional representation C, with X and C both influencing classification Y.
2.2 Structural Causal Model: Figure 3 illustrates feature-wise and class-wise meanings of C: channels respond to visual concepts, while features cluster around pre-training classes and their similarities.
2.2 Structural Causal Model: Conventional P(Y|X) can encode spurious correlations through D, so ideal FSL should target the true causal effect of X on Y using P(Y|do(X)).
2.3 Causal Intervention via Backdoor Adjustment: Many-shot learning approximates P(Y|X) with P(Y|do(X)), whereas few-shot sampling breaks this approximation because sample identity creates additional dependencies.
2.3 Causal Intervention via Backdoor Adjustment: Backdoor adjustment estimates P(Y|do(X)) by observing and stratifying the confounder D, but instantiating D is difficult when a third-party pre-trained network hides its training dataset.

3 Interventional Few-Shot Learning

IFSL implements backdoor adjustment by stratifying pre-trained knowledge through feature dimensions, pre-training classes, or both. These adjustments approximate the interventional objective while reducing computation and preserving applicability when only feature representations are available.

IFSL exploits two properties of pre-trained networks: feature dimensions carry semantic meaning, and pre-training commonly uses classification outputs.
Feature-wise Adjustment: Feature-wise adjustment partitions feature dimensions into n disjoint subsets and uses active dimensions within each subset to define pre-trained-knowledge strata.
Feature-wise Adjustment: The feature-wise classifier receives the selected feature dimensions [x]c, with a uniform prior P(di)=1/n over adjusted feature strata.
Feature-wise Adjustment: Feature-wise adjustment is always applicable because the feature representation x can always be obtained from the pre-trained network.
Class-wise Adjustment: Class-wise adjustment uses each pre-training class as a stratum and represents an input through the pre-trained class probability weighted by that class’s mean feature.
Class-wise Adjustment: The class-wise implementation concatenates x with g(x,di), assumes a uniform prior over pre-training classes, and forms the overall adjustment from these strata.
The method assumes linear combinations of feature dimensions or classes; otherwise adjustment requires prohibitive O(2^n) sampling, while NWGM reduces forward-pass consumption.
Combined Adjustment: The combined adjustment applies feature-wise adjustment after class-wise adjustment to obtain finer-grained stratification.

4 Related Work

Prior Few-Shot Learning work spans diverse adaptation, representation, transfer, and transductive methods, while negative transfer studies examine harmful source-to-target transfer. This paper distinguishes its causal perspective by treating pre-training as a confounder and applying that view to improve FSL methods.

Few-Shot Learning methods include fine-tuning, initialization optimization, parameter generation, feature-space learning, feature transfer, and transductive learning.
This work approaches Few-Shot Learning causally by identifying pre-training as a confounder and using that perspective to improve existing FSL methods.
Negative transfer describes source-domain learning that contributes negatively to target-domain performance.
Causal inference has been applied across computer-vision tasks including image classification, imitation learning, long-tailed recognition, and semantic segmentation.

5 Experiments

Experiments evaluate IFSL across standard, hardness-specific, visual-semantic, and cross-domain settings using multiple datasets, backbones, and FSL methods. IFSL consistently improves baselines, with especially large gains in lower-shot and cross-domain evaluations.

Datasets and Settings: Experiments use miniImageNet, tieredImageNet, and cross-domain CUB, with ResNet-10 and WRN-28-10 backbones and both fine-tuning and meta-learning methods.CUB models are trained on miniImageNet and evaluated across the domain gap.
Hardness-Specific Accuracy: IFSL improves fine-tuning and meta-learning consistently across query hardness, while backbone behavior differs between the two paradigms on harder queries.For fine-tuning, ResNet-10 becomes superior to WRN-28-10 as queries become harder; for meta-learning, deeper backbones remain better.
CAM-Acc and Visualization: IFSL increases attention to objects and improves CAM-Acc, suggesting that predictions rely more on correct visual semantics, although extremely small objects remain failure cases.The complete CAM-Acc results report similar or better performance across settings.
Cross-Domain Generalization: On cross-domain miniImageNet→CUB evaluation, IFSL brings consistent improvements averaging 1.94% of accuracy and improves both inductive and transductive methods.Improvements are larger for the inductive linear classifier than for transductive SIB, while SIB still benefits.

6 Conclusions

IFSL addresses the deficiency that pre-training can act as a confounder in Few-Shot Learning by modeling the relevant causalities and applying backdoor adjustment. It improves baselines across query hardness while providing a causal approximation to many-shot learning.

IFSL models causalities among pre-trained knowledge, few-shot samples, and class labels through a structural causal model.
Three practical IFSL implementations use backdoor adjustment to counteract the confounding effect of pre-training.
IFSL improves all baselines across query hardness while addressing the performance harm caused by pre-training.
IFSL offers a causal explanation for its effectiveness by approximating many-shot learning.
Future work will explore other observational intervention algorithms and counterfactual reasoning for domain-transfer few-shot settings.

8 Broader Impact

Few-Shot Learning supports applications where labeled data is difficult or expensive to obtain. The paper connects these applications to medical analysis, e-commerce recommendation, and industrial robotics.

Few-Shot Learning helps deploy machine-learning models when labeled data is difficult or expensive to obtain.
Applications include few-shot drug discovery and medical imaging analysis.
Applications also include cold-start item recommendation and few-shot reinforcement learning for industrial robots.

Supplementary Material for Interventional Few-Shot

The supplementary material organizes technical analyses, derivations, algorithms, and implementation details across Sections A.1–A.5.

Section A.1: Section A.1 analyzes many-shot learning through instrumental variables and its convergence toward the true causal effect.
Section A.2: Section A.2 derives the backdoor-adjustment formula used in Eq. (1).
Section A.3: Section A.3 derives the NWGM approximation used in Eqs. (3) and (4).
Section A.4: Section A.4 provides algorithms for adding IFSL to fine-tuning and meta-learning.
Section A.5: Section A.5 contains implementation details for pre-training and related procedures.

A.1 Instrumental Variable

The supplementary analysis uses instrumental variables to connect many-shot learning with the true causal effect. It explains how sampling ID can identify the effect of sample features on labels despite confounding.

In the many-shot causal graph, sampling ID I serves as an instrumental variable for X →Y.
d-separation determines conditional independence by checking whether nodes block every path between two variables.
An instrumental variable must be independent of Y after manipulating X while remaining associated with X in the original graph.
In the few-shot SCM, additional paths prevent sampling ID from satisfying the instrumental-variable criteria.
Regression between X and Y is confounded, whereas the ratio rIY/rIX recovers the true causal effect in the simplified linear case.

A.2 Derivation of Backdoor Adjustment for the Proposed Causal Graph

The appendix derives the backdoor adjustment for the proposed causal graph using do-calculus, transforming the desired interventional distribution into an expression over observed variables and stratifications. The derivation uses the three do-calculus rules and exploits the deterministic relation c = g(x, d).

Rule 1 inserts or deletes observations, Rule 2 exchanges actions and observations, and Rule 3 inserts or deletes actions.These are the three rules introduced for arbitrary disjoint node sets in the causal directed acyclic graph.
The derivation applies do-calculus to obtain the desired interventional distribution P(Y |do(X = x)) from the causal graph.The appendix states that the derivation uses the three do-calculus rules.
The resulting adjustment sums P(Y |X = x, D = d, C = c)P(C = c|X = x, D = d)P(D = d) over stratifications and feature values.This expression is given as Eq. (A11).
Because C is deterministically given by c = g(x, d), the summation over C reduces to a single probability measure.This reduction produces the expression shown in Eq. (A12).
The derivation uses D ⊥⊥X in the manipulated graph for Rule 3 and (Y ⊥⊥X|D) in the manipulated graph for Rule 2.These conditional independences justify the corresponding transformations in Eqs. (A9) and (A10).

A.3 Derivation of NWGM Approximation

The appendix derives the NWGM approximation used for IFSL by first moving expectation through the softmax and then through classifier functions. The second step is exact for linear and unit-vector cosine classifiers, while the general cosine case is approximate and k-NN is related to a linear parameterization.

NWGM approximates the expectation of softmax classifier outputs after applying the backdoor adjustment over stratifications D with prior P(d).Here c = g(x, d) is concatenated to x, and D represents the stratifications.
Linear classifier: For a linear classifier, f(x ⊕c) = W1x + W2c, allowing the expectation to move inside the classifier because x is shared across stratifications.The bias term is omitted because it does not affect the analysis, and E_d[x] = x.
Cosine classifier: For a cosine classifier with unit-vector x and c, the classifier reduces to a linear combination of x-only and c-only terms, yielding σ(E f(·)) = σ(f(E ·)).When x and c are not unit vectors, moving expectation inside the cosine classifier becomes an approximation.
k-NN classifier: For k-NN, class centroids are computed from support-set means and squared Euclidean logits are equivalent to a linear classifier with a particular parameterization.The implementation uses the nearest centroid for prediction, corresponding to 1-NN.
The approximation is derived as a two-stage transformation: E[σ(f(·))] ≈σ(E[f(·)]) followed by σ(E[f(·)]) ≈σ(f(E[·])).The first stage uses the Normalized Weighted Geometric Mean approximation, and the second analyzes classifier forms.

A.4 Algorithms for Fine-tuning and Meta-Learning with IFSL

The paper integrates IFSL into fine-tuning and meta-learning by replacing ordinary prediction with backdoor-adjusted intervention while retaining each framework’s training procedure. Experiments use frozen pretrained backbones, several classifier and meta-learning settings, and supplementary evaluations across accuracy and domains.

Fine-tuning: IFSL is added to fine-tuning by selecting an adjustment strategy that specifies D, g(X, D), P(Y |X, D, C), and P(D).The adjusted prediction is then used to compute loss and update classifier parameters.
Meta-learning: Meta-learning retains episode-based optimization: θ is initialized and fine-tuned on support data, queries are predicted with Pφ(y|do(x); θ), and φ is updated from query loss.Algorithm 2 incorporates Algorithm 1 within each sampled episode.
Fine-tuning: Fine-tuning with IFSL computes c = g(x, d), obtains P(Y |xi, c, d; θ) and P(d), predicts with P(y|do(x); θ), and updates θ.These steps are summarized in Algorithm 1.
Backbones and training: The experiments pre-train ResNet-10 or WRN-28-10 feature extractors before fine-tuning or meta-learning, using supervised training on dataset train splits.The backbones are trained from scratch with stochastic gradient descent and cross-entropy loss, with augmentation and early stopping.
Classifier settings: Fine-tuning experiments evaluate linear, cosine, and k-NN classifiers, with k-NN predicting from average support-set feature centroids.Linear and cosine classifier definitions and the nearest-centroid k-NN implementation are specified in the appendix.
Meta-learning and evaluation: The evaluation includes MAML, MTL, LEO, Matching Net, and SIB integrations, alongside supplementary conventional, hardness-specific, CAM, and cross-domain accuracy results.Table A1 reports accuracy and 95% confidence intervals for three adjustment implementations: feature-wise, class-wise, and combined.