Source-linked AI summary
Self-Supervised Generalisation with Meta Auxiliary Learning
Shikun Liu, Andrew J. Davison, Edward Johns
TL;DR
Auxiliary learning can improve generalisation but traditionally requires manually chosen or labelled auxiliary tasks. MAXL learns auxiliary labels from primary-task supervision through coupled label-generation and multi-task networks. Across image-classification evaluations, it outperforms single-task learning and auxiliary-label baselines while remaining competitive with human-defined labels.
Problem
Auxiliary learning typically requires domain knowledge and labelled auxiliary data, while unsupervised alternatives provide only a limited set of auxiliary tasks.
Method
MAXL uses a label-generation network and a multi-task network, updating auxiliary labels according to the multi-task network’s primary-task performance.
Results
MAXL outperforms single-task learning across seven image datasets, several auxiliary-label baselines, and is competitive with manually defined auxiliary labels.
Takeaways & Limitations
MAXL’s self-supervised auxiliary-label generation removes the need for manual auxiliary-task labelling in the evaluated image-classification setting.
Takeaways & Limitations
Preliminary experiments on predicting arbitrary vectors produced inconclusive results, leaving broader auxiliary-task regression capabilities unresolved.
Abstract
from arXiv · showhide
Learning with auxiliary tasks can improve the ability of a primary task to generalise. However, this comes at the cost of manually labelling auxiliary data. We propose a new method which automatically learns appropriate labels for an auxiliary task, such that any supervised learning task can be improved without requiring access to any further data. The approach is to train two neural networks: a label-generation network to predict the auxiliary labels, and a multi-task network to train the primary task alongside the auxiliary task. The loss for the label-generation network incorporates the loss of the multi-task network, and so this interaction between the two networks can be seen as a form of meta learning with a double gradient. We show that our proposed method, Meta AuXiliary Learning (MAXL), outperforms single-task learning on 7 image datasets, without requiring any additional data. We also show that MAXL outperforms several other baselines for generating auxiliary labels, and is even competitive when compared with human-defined auxiliary labels. The self-supervised nature of our method leads to a promising new direction towards automated generalisation. Source code can be found at https://github.com/lorenmt/maxl.
1 Introduction
Auxiliary learning can improve primary-task generalisation, but conventional approaches require manually selected or labelled auxiliary tasks. MAXL learns auxiliary labels from primary-task labels and couples this process with multi-task training.
- Motivation: Auxiliary learning improves primary-task generalisation by sharing features learned from additional tasks.The auxiliary task is included to assist the primary task rather than to achieve high performance independently.
- Motivation: Supervised auxiliary learning requires domain knowledge and labelled auxiliary data, while unsupervised alternatives offer a limited set of potentially useful tasks.The proposed framework aims to combine supervised flexibility with unsupervised avoidance of manual auxiliary labels.
- MAXL: MAXL automatically discovers auxiliary labels using only the labels available for the primary task.The method treats defining an auxiliary task as defining its labels.
- MAXL: MAXL trains a multi-task network alongside a label-generation network whose updates use primary-task performance to improve future auxiliary labels.The two networks are coupled end-to-end through a meta-learning objective.
- Results: MAXL outperforms single-task learning across seven image datasets, beats several auxiliary-label baselines, and remains competitive with manually defined labels.These comparisons use the same labelled data for MAXL and single-task learning, and MAXL does not use the manually defined auxiliary labels.
2 Related Work
The related work positions MAXL at the intersection of multi-task learning, auxiliary learning, and meta learning. These areas differ mainly in which tasks matter and whether the learning procedure, representations, or task specifications are being learned.
- Multi-task & Transfer Learning: Multi-task learning trains related tasks simultaneously to learn shared representations that improve individual-task performance.The shared knowledge is encoded in feature representations across tasks.
- Auxiliary Learning: Auxiliary learning differs from multi-task learning because only the primary task requires high test accuracy; auxiliary tasks support its generalisation.The related-work discussion describes auxiliary supervision as one route to assisting primary tasks.
- Connection to MAXL: MAXL connects these areas by learning auxiliary-task labels so that auxiliary training supports the primary task.This framing combines auxiliary learning’s primary-task objective with meta-learning’s emphasis on learning how learning proceeds.
- Meta Learning: Meta learning aims to induce the learning algorithm itself, including update rules, optimisers, hyperparameters, or initialisations.The cited work spans learned update rules, optimiser design, hyperparameter selection, and few-shot initialisation.
3 Meta Auxiliary Learning
MAXL automatically generates auxiliary labels and trains them alongside the primary task using two interacting networks. Its meta objective updates label generation according to primary-task performance, with hierarchical masking and entropy regularisation supporting informative auxiliary tasks.
- Task formulation: MAXL defines the auxiliary task as subclass labelling within a two-level hierarchy, with each primary class associated with auxiliary classes.The framework focuses on classification for both tasks and can use soft assignments for generated auxiliary labels.
- Network design: The multi-task network learns the primary and auxiliary tasks, while the label-generation network produces the auxiliary labels from the same inputs.The multi-task network uses shared features with task-specific output layers, and its parameters receive losses from both tasks.
- Optimisation: Each training epoch alternates between training the multi-task network on generated auxiliary labels and updating label generation to maximise primary-task performance.The label-generation update uses the multi-task network’s primary-task performance, creating a meta-learning interaction with a second derivative.
- Training objectives: Focal loss with focusing parameter γ = 2 is used for both primary and auxiliary tasks, emphasizing incorrectly predicted labels.The authors report improved performance compared with regular cross-entropy during experimental evaluation.
- Hierarchical predictions: Mask SoftMax restricts generated auxiliary-label predictions to classes associated with the ground-truth primary class.For hierarchy ψ = [2, 2], the mask permits each primary class to select from its own associated auxiliary classes; no domain knowledge is required to define the hierarchy.
- Regularisation: Entropy regularisation discourages auxiliary-label collapse by encouraging the label-generation network to use the full auxiliary-class prediction space.The entropy loss is defined through the divergence between predicted auxiliary-label distributions and a uniform distribution.
4 Experiments
Experiments evaluate MAXL across seven image datasets and multiple baselines, testing whether automatically generated auxiliary labels improve primary-task generalisation without additional labelled data. Results show consistent gains over single-task learning, strong performance against auxiliary-label baselines, and qualitative evidence that MAXL learns useful but often non-human-interpretable representations.
- Experimental setup: MAXL is evaluated on seven image-classification datasets using single-task learning and auxiliary-label-generation baselines.The experiments use three network architectures and varied hierarchies to test robustness without dataset-specific hierarchy tuning.
- Comparison to Single Task Learning: MAXL consistently outperforms single-task learning across all six reported datasets while using exactly the same training data.It also outperforms single-task learning across almost all tested hierarchy values ψ.
- Comparison to Auxiliary Label Generation Baselines: On CIFAR-100, MAXL outperforms Single Task, Random, and K-Means, despite all methods using exactly the same data.MAXL performs similarly to Human, which uses manually defined auxiliary labels; K-Means also requires significantly longer training because clustering runs after each iteration.
- Understanding the Utility of Auxiliary Labels: MAXL produces high cosine similarity between auxiliary and primary gradients throughout training across tested numbers of auxiliary classes.Fixed-label baselines peak early and then decline, while K-Means varies with the number of auxiliary classes.
- Understanding the Utility of Auxiliary Labels: Positive cosine similarity throughout training guarantees convergence of a standard shared-feature gradient update to a local minimum under a small learning rate.The authors caution that the optimal cosine similarity value is unknown.
- Visualisations of Generated Knowledge: t-SNE visualisations show better primary-class separation for MAXL and Human than for Single Task on two CIFAR-100 hierarchies.The comparison uses the final feature layer of the multi-task network.
- Visualisations of Generated Knowledge: Only part of MAXL’s generated auxiliary labels has human-understandable structure, while many groups lack obvious visual or semantic similarities.The authors argue this is consistent with labels targeting aspects of reasoning that the primary task struggles to learn, which may not be human-interpretable.
- Visualisations of Generated Knowledge: The generated auxiliary knowledge is not deterministic: top predicted candidates differ when the network is retrained from scratch.The authors speculate that different runs may reach different local optima rather than reproduce a single human-defined hierarchy.
5 Conclusion & Future Work
MAXL generates auxiliary labels without human-defined tasks and improves primary-task performance in image classification. Its broader use beyond subclass image classification remains an open direction, with preliminary regression experiments inconclusive.
- MAXL generates auxiliary labels that improve primary-task performance in a multi-task setup without relying on domain knowledge or human-defined auxiliary tasks.
- On multiple image-classification datasets, MAXL significantly outperforms auxiliary-label baselines and remains competitive with manually constructed human labels.
- Preliminary experiments predicting arbitrary vectors produced inconclusive regression results, leaving generic auxiliary tasks beyond subclass image classification unresolved.
A 4-level CIFAR-100 Dataset
Table 2 presents the CIFAR-100 dataset organized as a 4-level hierarchy.
- CIFAR-100 is represented in a 4-level hierarchy.
B Training Strategies
Training uses SGD-based optimization for both MAXL networks, with schedules and regularization differing between the multi-task and label-generation networks.
- The multi-task network uses SGD with dataset-dependent learning rates and schedules, including cosine annealing for six datasets.For 4-level CIFAR-100, the learning rate is 10^-2 and halves every 50 epochs over 200 epochs; the other datasets use learning rate 0.1 with momentum 0.9 and weight decay 5·10^-4.
- The label-generation network uses learning rate 10^-3 and weight decay 5 · 10^-4 to help prevent class collapse, with entropy-loss weight λ = 0.2.
C Further Analysis on the Collapsing Class Problem
The analysis compares CIFAR-100 performance with and without entropy loss across hierarchy structures, alongside auxiliary-label utilization.
- Entropy loss with λ = 0.2 makes MAXL utilize the entire auxiliary-label space and improves performance over using no entropy loss.The comparison reports test accuracy and the percentage of auxiliary labels assigned by the label-generation network.
- Table 3 compares accuracies for the 4-level CIFAR-100 dataset with and without entropy loss.
D Cosine Similarity on CIFAR-100 Dataset
Figure 7 measures cosine similarity between the auxiliary-loss and primary-loss gradients on the multi-task network’s shared representation.
- Figure 7 measures cosine similarity between auxiliary-loss and primary-loss gradients on the shared representation.
E Negative Results
The reported experiments identify several design choices that limited MAXL’s performance, including loss selection, Mask SoftMax, auxiliary output type, update scheduling, and task setting.
- Standard cross-entropy produced worse performance than focal loss.The focal loss focuses on incorrectly predicted labels.
- MAXL without Mask SoftMax achieved performance similar to single-task learning.
- MAXL producing an auxiliary latent vector for regression achieved performance similar to single-task learning.
- Updating the multi-task and label-generation networks in the same iteration performed worse than updating them independently for multiple iterations per epoch.
- MAXL provided only marginal benefit on semantic segmentation tasks.
- Updating the label-generation network from unseen validation performance was less beneficial than updating it from the same training data.