PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification

Jiangtao Xie, Fei Long, Jiaming Lv, Qilong Wang, Peihua Li

arXiv:2204.04567v1cs.CVcs.LGstat.ML

TL;DR

Few-shot classification must generalize to unseen classes from very few examples, while existing methods either overlook joint feature distributions or incur high computational cost. DeepBDC measures the discrepancy between joint characteristic functions and products of marginals using a modular BDC layer, instantiated in two frameworks. Across six benchmarks, both instantiations achieve superior performance and establish new state-of-the-art results.

  • Problem

    Few-shot classification requires distinguishing unseen classes from very few examples, while prior methods either model only marginal distributions or are computationally expensive when modeling joint distributions.

  • Method

    DeepBDC learns image representations by measuring the discrepancy between joint characteristic functions and products of marginals through a modular, efficient BDC layer.

  • Results

    Both Meta DeepBDC and STL DeepBDC achieve superior performance and set new state-of-the-art results across six few-shot classification benchmarks.

  • Takeaways & Limitations

    DeepBDC provides a flexible representation-learning approach suitable for both episodic meta-learning and non-episodic simple transfer learning.

  • Takeaways & Limitations

    The BDC definition assumes X and Y have finite first moments, and the practical i.i.d. assumption may not hold.

Abstract

from arXiv · show

Few-shot classification is a challenging problem as only very few training examples are given for each new task. One of the effective research lines to address this challenge focuses on learning deep representations driven by a similarity measure between a query image and few support images of some class. Statistically, this amounts to measure the dependency of image features, viewed as random vectors in a high-dimensional embedding space. Previous methods either only use marginal distributions without considering joint distributions, suffering from limited representation capability, or are computationally expensive though harnessing joint distributions. In this paper, we propose a deep Brownian Distance Covariance (DeepBDC) method for few-shot classification. The central idea of DeepBDC is to learn image representations by measuring the discrepancy between joint characteristic functions of embedded features and product of the marginals. As the BDC metric is decoupled, we formulate it as a highly modular and efficient layer. Furthermore, we instantiate DeepBDC in two different few-shot classification frameworks. We make experiments on six standard few-shot image benchmarks, covering general object recognition, fine-grained categorization and cross-domain classification. Extensive evaluations show our DeepBDC significantly outperforms the counterparts, while establishing new state-of-the-art results. The source code is available at http://www.peihuali.org/DeepBDC

1. Introduction

Few-shot classification must learn to distinguish previously unseen classes from very few examples, making representation and similarity learning difficult. DeepBDC addresses this gap by modeling joint feature dependencies with an efficient, modular BDC layer and achieves superior performance across six benchmarks.

  • Few-shot classification adapts classifiers to unseen classes using only a very limited number of examples, which can provide insufficient knowledge and promote overfitting.
  • Metric-based methods learn representations from similarity between query and support images, but modeling high-dimensional feature distributions is difficult.
  • Moments-based methods model marginal distributions, whereas joint-distribution methods such as DeepEMD provide richer dependency modeling but can be computationally expensive.
  • DeepBDC measures the discrepancy between joint characteristic functions and products of marginals, capturing nonlinear relations and independence through BDC.
  • BDC is formulated as a modular, efficient pooling layer that outputs BDC matrices, whose inner products measure image similarity.
  • The method is instantiated as Meta DeepBDC and STL DeepBDC, and experiments on six benchmarks report superior performance and new state-of-the-art results.

2. Related Works

Related work includes few-shot representations based on distribution modeling and frameworks based on episodic meta-learning or non-episodic simple transfer learning. DeepBDC differs by using Brownian Distance Covariance for representation learning.

  • Distribution-based few-shot methods use first moments, second moments, Gaussian distributions, or discrete probabilities to represent images.
  • DeepBDC differs from existing representation-learning approaches by using Brownian Distance Covariance in the few-shot regime.
  • Meta-learning versus simple transfer learning: Meta-learning uses episodic N-way K-shot tasks and transfers knowledge across meta-training and meta-testing classes.
  • Meta-learning versus simple transfer learning: Simple transfer learning trains on the whole meta-training set with standard cross-entropy and does not rely on episodic training.
  • Few studies have used BDC in machine learning or computer vision, with prior work including a BDC-based dimension-reduction method outside deep learning.

3. Proposed Method

DeepBDC measures dependence between embedded feature sets using Brownian Distance Covariance, then turns the decoupled metric into a modular pooling layer for few-shot representations. It supports both episodic meta-learning and simple transfer-learning instantiations.

  • 3.1. Brownian Distance Covariance (BDC): Brownian Distance Covariance measures dependence through the discrepancy between a joint characteristic function and the product of its marginals.The metric is nonnegative, vanishes exactly under independence, captures linear and nonlinear dependence, and has stated translation, rotation, and scaling properties.
  • 3.1. Brownian Distance Covariance (BDC): For i.i.d. discrete observations, BDC uses Euclidean distance matrices and double-centers them to obtain BDC matrices whose vectorized upper triangles form an inner product.The construction computes pairwise distances, subtracts row, column, and overall means, and exploits matrix symmetry.
  • 3.2. Formulation of DeepBDC as a Pooling Layer: DeepBDC is a parameter-free spatial pooling layer using standard matrix operations, while a 1×1 convolution reduces channels because BDC matrix size grows quadratically with channel count.The layer is described as modular, compatible with varying architectures and classification frameworks, and suitable for GPU parallelization.
  • 3.2. Formulation of DeepBDC as a Pooling Layer: DeepBDC reshapes an embedded image tensor into a matrix, treats spatial feature vectors as observations, and computes a BDC matrix through distance and centering operations.The method notes that the i.i.d. assumption may not hold and compares spatial with channel pooling options.
  • 3.3. Instantiating DeepBDC for Few-shot Learning: Meta DeepBDC combines BDC representations with ProtoNet-style prototype distances and episodic task sampling for meta-learning.Support-class prototypes average BDC matrices, and classification uses a softmax over distances with a learnable scaling parameter.
  • 3.3. Instantiating DeepBDC for Few-shot Learning: STL DeepBDC trains a conventional classifier on all meta-training classes, then uses the network as an embedding model and trains logistic regression during meta-testing.The paper presents Meta DeepBDC and STL DeepBDC as two framework-specific instantiations of the same pooling-based representation.

4. Experiments

Experiments evaluate Meta DeepBDC and STL DeepBDC across standard, fine-grained, and cross-domain few-shot settings, including ablations, latency comparisons, and state-of-the-art benchmarks. Both instantiations generally outperform competing methods, with spatial pooling and moderate dimensionality providing favorable accuracy–efficiency trade-offs.

  • Experimental settings: Experiments cover six few-shot datasets spanning general object recognition, fine-grained categorization, and cross-domain classification.The study uses Meta DeepBDC and STL DeepBDC, with ProtoNet and non-episodic transfer-learning blueprints.
  • Ablation study: Meta DeepBDC improves over ProtoNet across channel dimensions while adding only a small latency increase.BDC matrices scale quadratically with reduced channel dimension d, motivating the 1×1 convolution used in the ablation.
  • Ablation study: STL DeepBDC with d = 128 achieves the best STL result, while larger d increases overfitting risk and lowers accuracy.Logistic regression outperforms softmax and linear SVM, though it has higher latency than SVM.
  • Ablation study: Spatial pooling outperforms channel pooling: Meta DeepBDC reaches 62.55±0.45/78.88±0.32 and STL DeepBDC 63.95±0.45/79.45±0.32 for 1-shot/5-shot channel pooling.The comparison is made under the same miniImageNet 5-way setting; the authors note that i.i.d. assumptions may not hold for either pooling choice.
  • Comparison with counterparts: Both DeepBDC instantiations outperform distribution-modeling counterparts by large margins, consistent with stronger joint-dependency modeling.They are slightly slower than ProtoNet and CovNet, while DeepEMD is approximately 2 times and 50 times slower for 1-shot and 5-shot tasks.
  • Benchmark comparisons: On miniImageNet 5-shot, Meta DeepBDC and STL DeepBDC achieve 68.67±0.39 and 69.07±0.39, versus ProtoNet’s 55.96±0.38.On another reported 5-shot setting, the corresponding accuracies are 54.61±0.37 and 58.09±0.36, versus ProtoNet’s 46.30±0.36.

5. Conclusion

The paper concludes that DeepBDC learns few-shot image representations by comparing joint feature distributions with products of marginals through a modular network layer. Its two frameworks perform better than counterparts and achieve new state-of-the-art results across general, fine-grained, and cross-domain tasks.

  • Conclusion: DeepBDC measures the discrepancy between joint distributions of query and support features and the product of their marginals.The method is formulated as a modular, efficient layer usable in both episodic meta-learning and non-episodic transfer learning.
  • Conclusion: Extensive experiments report stronger performance than counterparts and new state-of-the-art results across multiple few-shot task types.The evaluated categories include general object recognition, fine-grained classification, and cross-domain classification.

Supplementary Material

The supplement adds implementation details, further ablations and comparisons, and experiments showing that BDC can characterize non-linear dependence.

  • Supplementary material: Supplementary experiments provide implementation details, additional ablations and comparisons, and evidence that BDC characterizes non-linear dependence.The supplement covers DeepBDC and counterpart implementations.

S1 Implementations

The implementations cover multiple few-shot benchmarks, backbone architectures, training procedures, and evaluation protocols. DeepBDC is instantiated through both meta-learning and simple transfer-learning settings.

  • Datasets: The experiments span miniImageNet and tieredImageNet for general recognition, CUB for fine-grained classification, and three miniImageNet-based cross-domain tasks.The cross-domain tasks evaluate transfer to CUB, Aircraft, and Cars.
  • Backbone network: The study uses ResNet-12 and ResNet-18 backbones, with additional higher-capacity ResNet-34 variants.Input resolution is 84×84 for ResNet-12 and 224×224 for ResNet-18 and ResNet-34.
  • Training: Training applies standard image augmentation, SGD with momentum 0.9, weight decay 5e-4, and DropBlock regularization for ResNet-12.Epoch counts and learning-rate schedules are tuned separately across benchmarks.
  • Training: Meta DeepBDC uses episodic training after pre-training, sampling 5-way 1-shot or 5-shot tasks with 16 query images.Each task contains five classes and either one or five support images per class.
  • Training and evaluation: STL DeepBDC trains a multi-way classifier with cross-entropy, then uses sequential self-distillation to obtain embedding models without episodic training.A linear classifier is trained separately for each evaluation episode.
  • Evaluation: Evaluation reports mean accuracy over 2,000 sampled 5-way episodes with 95% confidence intervals.Meta DeepBDC requires no additional testing training, whereas STL DeepBDC trains a linear classifier per episode.

S2 Implementation of the Counterparts

The counterpart implementations reproduce public methods while adding dimension reduction or fusion adjustments where needed for comparison. All methods use compatible backbone and initialization settings.

  • ProtoNet: ProtoNet is evaluated using the publicly available implementation of Chen et al.The implementation is used as released by its authors.
  • ADM: ADM adds 1×1 convolutional dimension reduction and is combined with DN4 using separately learnable branch scaling parameters.Negative KL-divergence and Image-to-Class scores are separately normalized before fusion.
  • CovNet: CovNet uses 1×1 convolutional dimension reduction and the inner product for 5-way 1-shot classification instead of the Frobenius norm.The inner product is used because the Frobenius norm produced poor results in that setting.
  • Shared settings: The compared methods remove the final backbone down-sampling and initialize episodic training from pre-trained model weights.This follows the stated previous experimental practice.

S3 Additional Experiments

Additional experiments examine prototype construction, model capacity, feature-map count, latency, and channel-dimension effects. They also compare DeepBDC with baselines under matched implementation settings.

  • Prototype construction: For 5-shot Meta DeepBDC, averaging support-image BDC matrices is compared with averaging features first or concatenating features before BDC computation.The alternatives achieve reported accuracies of 82.36% and additional values beyond the supplied passage.
  • Effect of Higher Capacity Models: Higher-capacity ResNet-34 models improve accuracy over ResNet-12 on miniImageNet across methods and settings.ProtoNet and Good-Embed show improvements above 1 percentage point for 1-shot tasks, while DeepBDC retains large margins over its baselines.
  • Effect of Feature Number on DeepBDC: Changing feature count has a minor effect on STL DeepBDC, while Meta DeepBDC improves slightly at 100 features before declining at 441 features.ProtoNet and Good-Embed achieve their individual best results at 100 and 25 features, respectively.
  • Latency: ProtoNet and CovNet have the fastest meta-training latency, while Meta DeepBDC is somewhat slower and ADM has much larger meta-training latency than DeepBDC.The comparison uses ResNet-12 on miniImageNet for 5-way classification.
  • Channel number: Second-moment methods require dimension reduction because their representations grow quadratically with channel number d.The re-implementation improves CovNet and ADM and supports dimension reduction as an important design factor for these methods.
  • Channel number: ADM and CovNet reach peak accuracies at d = 196 (82.05) and d = 256 (82.02), respectively, then decline as d increases.At d = 640, both are only slightly more accurate than ProtoNet.

S4 Linear and Non-linear Relation Modeling

Brownian Distance Covariance is designed to capture both linear and nonlinear dependency between random variables. Simulated examples show that it detects nonlinear relationships that classical correlation can miss.

  • Simulated relations: The experiments simulate nonlinear bivariate relationships including W-shape, Diamond, Parabola, Two parabolas, Circle, Butterfly, and Heart.They also include seven kinds of linear relations.
  • Notation: Covariance and Brownian distance covariance provide the corresponding variance quantities when both arguments are the same variable.The notation identifies cov(X, X) and BDC(X, X) as variance and Brownian distance variance.
  • Nonlinear relations: For the nonlinear relations, classical correlation is zero while Brownian distance correlation characterizes the dependencies.This contrast is presented through scatter plots and corresponding correlation values.
  • Linear relations: Brownian distance correlation has similar capability to classical correlation for modeling the simulated linear relations.The supplied passage begins this comparison but does not provide the complete numerical results.
Loading 2204.04567v1…