Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes

James Requeima; Jonathan Gordon; John Bronskill; Sebastian Nowozin; Richard E. Turner

Source-linked AI summary

Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes

James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, Richard E. Turner

arXiv:1906.07697v2Jun 18, 2019stat.MLcs.LG

TL;DR

The paper addresses image classification across diverse new tasks and data regimes without costly retraining. It introduces CNAPS, a conditional neural process model whose classifier parameters are generated or modulated from the current task’s dataset. CNAPS achieves state-of-the-art performance on Meta-Dataset and supports fast adaptation and deployment in continual and active learning.

Problem
Image classifiers need to adapt to new tasks from diverse data distributions in low- and high-data regimes, while existing methods often assume homogeneous train and test tasks.
Method
CNAPS uses shared global classifier parameters, task-specific model parameters, and an adaptation network that computes task parameters from the task dataset.
Results
CNAPS achieves state-of-the-art performance on Meta-Dataset, improves over existing methods on seven of eight unseen datasets, and is at least 5 times faster than gradient-based optimization at test time.
Takeaways & Limitations
Trained CNAPS models can be deployed out-of-the-box for continual and active learning and remain competitive with state-of-the-art methods.
Takeaways & Limitations
Local parameters learned via optimization are computationally demanding, require expert oversight, and can over-fit in low-data regimes.

Abstract

from arXiv · show

The goal of this paper is to design image classification systems that, after an initial multi-task training phase, can automatically adapt to new tasks encountered at test time. We introduce a conditional neural process based approach to the multi-task classification setting for this purpose, and establish connections to the meta-learning and few-shot learning literature. The resulting approach, called CNAPs, comprises a classifier whose parameters are modulated by an adaptation network that takes the current task's dataset as input. We demonstrate that CNAPs achieves state-of-the-art results on the challenging Meta-Dataset benchmark indicating high-quality transfer-learning. We show that the approach is robust, avoiding both over-fitting in low-shot regimes and under-fitting in high-shot regimes. Timing experiments reveal that CNAPs is computationally efficient at test-time as it does not involve gradient based adaptation. Finally, we show that trained models are immediately deployable to continual learning and active learning where they can outperform existing approaches that do not leverage transfer learning.

1 Introduction

General-purpose image classifiers must adapt to heterogeneous tasks across low- and high-data regimes without costly retraining. CNAPS addresses this through learned task adaptation while balancing flexibility, computation, and over-fitting risk.

Existing meta-learning and few-shot classification often assume homogeneous training and test task distributions, requiring only minimal adaptation.
Adapting only the classifier head can under-fit heterogeneous tasks, whereas adapting the entire feature extractor increases capacity but risks computation costs and low-shot over-fitting.
Gradient-based adaptation is flexible but inefficient and prone to over-fitting, while amortized function mappings offer fixed-cost adaptation but may under-fit or require many training tasks.
CNAPS introduces a conditional neural process model class for multi-task classification with task adaptation learned from diverse datasets and tasks.
CNAPS uses a small set of task-specific parameters and a rich auto-regressive adaptation network to balance flexibility and robustness while avoiding under-fitting.

2 Model Design

CNAPS specializes conditional neural processes for multi-task image classification by generating task-specific classifier and feature-extractor parameters from each task’s context set. It combines flexible feature adaptation with parameter-efficient FiLM modulation and permutation-invariant weight generation for variable-class, variable-size tasks.

Classifier specification: CNAPS uses global classifier parameters θ alongside task-specific parameters ψτ generated from the task context set Dτ.The adaptation function ψφ(Dτ) produces local parameters, while θ and the adaptation-network parameters φ are learned globally.
Classifier specification: The global feature extractor fθ(x) feeds a task-specific affine classifier whose output dimension matches the task’s number of classes Cτ.The task-specific weights ψτ_w have shape R^df × Cτ, with df the feature dimension and Cτ the number of classes.
Feature-extractor adaptation: FiLM adapts convolutional feature maps by scaling and shifting them with task-specific parameters, providing expressive feature adaptation with relatively few parameters.CNAPS places FiLM layers after every convolutional layer in its ResNet18 implementation; the adapted parameters constitute fewer than 0.7% of the model parameters.
Adaptation-network design: Direct optimization of local parameters is computationally demanding, requires tuning oversight, and can over-fit with little data, whereas CNAPS predicts them through a comparatively cheap forward pass.The learned adaptation network sacrifices some optimization flexibility for automatic, computationally cheaper task adaptation.
Adaptation-network design: CNAPS generates classifier-weight columns independently from class-specific context points and uses permutation-invariant set functions to handle variable class counts and context-set sizes.The classifier operates on adapted feature representations rather than raw inputs, and the resulting set architecture can be incrementally updated for continual learning.
Feature-extractor adaptation: The feature-extractor adaptation network combines a global task representation with an auto-regressive representation carrying information about adaptations in shallower layers.The global representation is shared across layers, while deeper layers receive information computed from adapted activations of the previous convolutional block.

3 Model Training

CNAPS trains global classifier parameters and adaptation-network parameters in a staged procedure designed to preserve useful feature adaptation at test time. The adaptation network is trained by maximum likelihood using episodic context-target splits.

3 Model Training: Training separates global classifier pre-training from later adaptation-network training.The global parameters θ are trained first, while φ={φf,φw} is trained afterward for task adaptation.
3 Model Training: Joint likelihood training of θ and φ performed poorly because it provided little training signal for feature-extractor adaptation.When feature adaptation is trained across all tasks, ψτf receives little-to-no signal, harming generalization.
3 Model Training: Figure 6 organizes methods by task-specific parameter count and adaptation-mechanism complexity, contrasting flexibility, over-fitting risk, and computational efficiency.Amortized approaches lie toward greater computational efficiency, while gradient-based approaches offer flexibility with more tuning and over-fitting risk.
3 Model Training: Fixing θ during meta-training matches test-time deployment and dramatically reduces training time.At test time θ is fixed, so training with fixed θ simulates the intended scenario.
3 Model Training: The adaptation parameters φ are optimized by maximum likelihood over sampled task context-target splits.This produces the episodic training protocol used in the experiments and META-DATASET.

4 Related Work

CNAPS extends conditional neural processes to heterogeneous multi-task image classification while addressing trade-offs in adaptation capacity and test-time cost. It differs from related approaches through richer task adaptation and a less ad-hoc training strategy.

4 Related Work: The paper frames multi-task classification as directly modeling p(y∗|x∗,ψ(Dτ)) through classifier parameterization and task-data adaptation choices.This framework organizes prior methods by their local parameters and the function that computes them from training data.
4 Related Work: CNAPS applies the conditional neural process framework to heterogeneous classification datasets using a more complex model than the original CNP setting.The model supports multiple datasets and varying classification tasks rather than homogeneous, relatively simple problems.
4 Related Work: Compared with TADAM, CNAPS uses a richer adaptation network and avoids TADAM’s expensive, ad-hoc training procedure.TADAM adapts both feature extraction and classification but was evaluated mainly on simpler few-shot benchmarks.

5 Experiments and Results

Experiments evaluate CNAPS on few-shot classification, adaptation speed, continual learning, and active learning. Across these settings, the results support transfer to unseen datasets and learning scenarios, while ablations emphasize the value of feature-extractor adaptation.

Few Shot Classification: CNAPS improves significantly over existing methods on seven of eight unseen META-DATASET datasets.TEXTURES is the exception, where only seven test classes make accuracy highly sensitive to the class split.
Few Shot Classification: Removing the auto-regressive task representation degrades accuracy in most cases, while removing all feature-extractor adaptation causes drastic reductions.These ablations support adapting key network components for diverse tasks.
Few Shot Classification: On unseen datasets, CNAPS outperforms all competitive models except FINETUNE on TRAFFIC SIGNS.Removing feature-extractor adaptation significantly impairs performance, especially when the held-out dataset differs substantially from pretraining data.
Speed-Accuracy Trade-off: CNAPS generates FiLM parameters through an adaptation network and is compared with gradient-descent feature-extractor adaptation across processing time and shot counts.The comparison uses withheld datasets, 5-way tasks, and shot counts from 1 to 25; gradient descent uses 25 steps per point.
Continual Learning: CNAPS resists catastrophic forgetting and performs similarly to or better than Riemannian Walk in continual learning.The model was not trained explicitly for continual learning and was evaluated despite seeing fewer examples and no training exposure to the target datasets.
Complex Learning Scenarios: The continual- and active-learning results demonstrate out-of-the-box versatility and strong performance in new domains and learning scenarios.The authors frame these comparisons as demonstrations of transfer-enabled deployment rather than apples-to-apples comparisons.
Active Learning: CNAPS achieves higher average active-learning accuracy than prototypical networks and improves significantly over random acquisition.Prototypical networks do not show the same improvement over random acquisition.

6 Conclusions

The paper concludes that CNAPS is an automatic, fast, and flexible approach to multi-task classification, achieving strong benchmark performance and supporting continual and active learning.

CNAPS achieves state-of-the-art performance on the META-DATASET challenge.
CNAPS adapts classifier representations and parameters to diverse tasks without relying on costly gradient-based test-time adaptation.
CNAPS models are immediately deployable to continual and active learning scenarios, where they are competitive with state-of-the-art methods.
Adapting more parameters increases flexibility but also computation and over-fitting risk, whereas adapting only the classifier head can under-fit heterogeneous tasks.
The model is designed for both low- and high-data regimes by balancing adaptation flexibility with reliability.

C.1.3 Evaluation

Evaluation uses 600 test episodes per dataset with accuracy averaged across episodes and 95% confidence intervals. CNAPS is competitive even when trained only on ILSVRC-2012, while its learned representations separate diverse datasets at test time.

600 test episodes per dataset are used, with classification accuracy averaged across episodes and 95% confidence intervals computed.
CNAPS remains competitive on META-DATASET when trained on ILSVRC-2012 only.
The ILSVRC-only setting does not test adaptation to genuinely unseen datasets because the model never sees data outside θ pre-training.
The set encoder separates examples from diverse datasets, while FiLM generators produce dataset-specific feature-extractor parameters.
CIFAR10 and CIFAR100 show the only significant overlap in FiLM-parameter plots, consistent with their close relationship.

D.3 Joint Training of θ and φ

Experiments show that jointly training the global and adaptation parameters is unstable, whereas the proposed two-stage procedure substantially improves performance.

Joint training of θ and φ diverged in almost all attempted cases.
The two-stage training procedure greatly improves performance compared with joint training, including joint training with batch normalization in train mode.
CNAPS uses FiLM layers in series with convolutional layers, requiring 2C adaptation parameters per layer when the feature-channel count is C.
Parallel residual adapters require C^2 parameters per convolutional layer, compared with 2C for CNAPS.

E.1 ResNet18 Architecture details

The implementation uses a ResNet18 feature extractor and set-based adaptation networks that generate task-specific FiLM parameters and class-specific linear-classifier weights and biases.

E.1 ResNet18 Architecture details: ResNet18 is used as the feature extractor throughout the experiments.
E.1 ResNet18 Architecture details: The feature extractor is composed from ResNet18 basic and basic scaling blocks, with FiLM layers enabled in the implementation.
Adaptation networks: The set encoder maps each context set Dτ to a global representation zG.
Adaptation networks: Auto-regressive networks summarize activations from preceding blocks, and their representations are concatenated with zG before generating the next layer’s FiLM parameters.
Linear classifier adaptation: The linear-classifier adaptation network mean-pools adapted feature activations for each class before generating class-specific weights and biases.
Linear classifier adaptation: Separate networks generate classifier weights and biases, which are then used as a linear classification layer.

F Continual Learning Implementation Details

CNAPS supports continual learning by storing compact representations of observed training data and updating them as new class data arrives. The procedure has an order-dependence caveat because feature-extractor adaptation parameters use only the most recent training data.

F Continual Learning Implementation Details: CNAPS stores a compact representation for each class and updates it when new training data for that class is observed.The update uses a pooled representation of the new examples.
F Continual Learning Implementation Details: Each class representation is updated with a weighted average that combines its previous value with the pooled representation of M new examples.The update weights the old and new representations by their respective example counts, M and N.
F Continual Learning Implementation Details: At prediction time, the stored class representation is supplied to ψw(·) to produce the classification parameters for that class.The classifier is therefore conditioned on the stored training-data representation.
F Continual Learning Implementation Details: The procedure is not completely invariant to training-data order because feature-extractor adaptation parameters depend only on the most recent training data.The authors observed little within-dataset variance in adaptation parameters, but anticipate that the issue may require addressing for continual learning across multiple datasets.

G Additional Continual Learning Results

Additional experiments evaluate CNAPS through task-level continual-learning trajectories on Split MNIST and Split CIFAR100. CNAPS remains competitive with RWalk while using less prior data and exhibits resistance to forgetting through internal task representations.

G Additional Continual Learning Results: Figure 11 reports task-level continual-learning performance for CNAPS and RWalk across the five Split MNIST tasks.The figure compares multi-head and single-head settings, with varying numbers of observed examples for CNAPS.
G Additional Continual Learning Results: CNAPS is competitive with RWalk on Split MNIST despite seeing less data per task and not retraining with old data at every time step.The comparison concerns methods’ continual-learning behavior as tasks are observed over time.
G Additional Continual Learning Results: CNAPS is described as naturally resistant to forgetting because internal task representations preserve important information from earlier time steps.This explanation is given for the Split MNIST continual-learning results.
G Additional Continual Learning Results: CNAPS maintains similar results on the more difficult CIFAR100 setting and performs comparably to, or better than, RWalk.CNAPS was not trained on CIFAR100, whereas RWalk was explicitly trained for this task and uses previous-task samples at each time step.

H Additional Active Learning Results

Additional active-learning experiments cover all twenty held-out Omniglot languages. CNAPS improves performance and enables standard acquisition functions to improve data efficiency over random acquisition in almost all languages.

H Additional Active Learning Results: The additional Omniglot active-learning evaluation reports results across all twenty held-out languages.This extends earlier results that covered three held-out test languages.
H Additional Active Learning Results: In almost all held-out languages, CNAPS improves overall performance and makes standard acquisition functions more data-efficient than random acquisition.The result uses CNAPS’s predictive distribution to guide acquisition.
H Additional Active Learning Results: Prototypical Networks generally do not obtain the same acquisition benefit, with random acquisition performing as well as or better than predictive-distribution-based acquisition functions in most cases.The comparison is reported for the held-out Omniglot languages.