Source-linked AI summary
Pareto Self-Supervised Training for Few-Shot Learning
Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, Donglin Wang
TL;DR
Few-shot auxiliary learning must balance conflicting few-shot and self-supervised objectives, yet suitable loss weights are difficult to choose. PSST decomposes the problem into constrained multi-objective subproblems and explores a preference region favoring the main task. Experiments report state-of-the-art benchmark performance and gains in exploration efficiency and effectiveness.
Problem
Few-shot auxiliary learning shares parameters across few-shot and self-supervised tasks, but conflicting objectives make suitable weights for a single weighted-sum solution difficult to choose.
Source: §1 · p. 1Method
PSST decomposes few-shot auxiliary learning into constrained multi-objective subproblems with different trade-off preferences and explores the region where the main task performs best.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The authors conclude that PSST reduces task conflict in few-shot auxiliary learning.
Source: §5 · p. 8The preference-region criterion assumes that best overall performance occurs when all tasks are balanced.
Source: §3.3.2 · p. 5
Abstract
from arXiv · showhide
While few-shot learning (FSL) aims for rapid generalization to new concepts with little supervision, self-supervised learning (SSL) constructs supervisory signals directly computed from unlabeled data. Exploiting the complementarity of these two manners, few-shot auxiliary learning has recently drawn much attention to deal with few labeled data. Previous works benefit from sharing inductive bias between the main task (FSL) and auxiliary tasks (SSL), where the shared parameters of tasks are optimized by minimizing a linear combination of task losses. However, it is challenging to select a proper weight to balance tasks and reduce task conflict. To handle the problem as a whole, we propose a novel approach named as Pareto self-supervised training (PSST) for FSL. PSST explicitly decomposes the few-shot auxiliary problem into multiple constrained multi-objective subproblems with different trade-off preferences, and here a preference region in which the main task achieves the best performance is identified. Then, an effective preferred Pareto exploration is proposed to find a set of optimal solutions in such a preference region. Extensive experiments on several public benchmark datasets validate the effectiveness of our approach by achieving state-of-the-art performance.
1. Introduction
Few-shot auxiliary learning uses self-supervised tasks to support learning with scarce labels, but balancing conflicting task objectives with a single weighted sum is difficult. PSST uses constrained multi-objective subproblems and preferred Pareto exploration; experiments report state-of-the-art performance on several benchmark datasets.
1. Introduction
- Few-shot learning addresses settings where labeled samples are rare or expensive and machines struggle to generalize to new concepts from little supervision.The paper notes that training large models with few labeled samples can lead to overfitting or non-convergence.Source: §1 · p. 1
- Few-shot auxiliary learning shares parameters between few-shot and self-supervised tasks, but their objectives can conflict and experience-set loss weights make optimal trade-offs difficult to choose.Prior approaches seek one solution for all objectives, which may sacrifice the main task's performance [16].Source: §1 · p. 1
- PSST decomposes few-shot auxiliary learning into constrained multi-objective subproblems with different trade-off preferences, then explores the region where the main task performs best.The preferred exploration is intended to improve main-task performance and the efficiency and accuracy of exploration.Source: §1 · pp. 1–2
- PSST achieves state-of-the-art performance on several benchmark datasets.The experiments are reported as demonstrating better modeling of the trade-off between tasks.Source: §1 · p. 2
2. Related Work
Related work reviews few-shot learning approaches, self-supervised pretext tasks and auxiliary learning, and multi-objective optimization. It identifies the challenge of finding Pareto solutions that prioritize the main task in auxiliary learning.
2.1. Few-Shot Learning
- Few-shot learning methods are grouped as optimization-based, model-based, and metric-based approaches.Optimization-based methods adjust the optimization algorithm; model-based methods rely on designed models, while metric-based methods learn embeddings for distance-based classification.Source: §2.1 · p. 2
2.2. Self-Supervised Learning
- Self-supervised learning learns representations from unlabeled data through pretext tasks or contrasting augmented positive and negative pairs; it is also used as an auxiliary task for few-shot learning.Prior few-shot auxiliary approaches use rotation prediction and relative patch location [16] or jigsaw puzzles.Source: §2.2 · p. 2
2.3. Multi-Objective Optimization
- Multi-objective optimization seeks Pareto solutions with different trade-offs, but prior approaches focus on equally important tasks, leaving main-task-prioritized Pareto search challenging for auxiliary learning.Population-based evolutionary methods are described as unsuitable for large-scale gradient-based multi-task learning, while gradient-based methods address that setting.Source: §2.3 · pp. 2–3
3. Pareto Self-Supervised Training
PSST combines few-shot classification with self-supervised tasks, then treats their losses as a multi-objective problem rather than a weighted sum. It identifies a preference region favoring the main task and explores Pareto solutions within that region.
3.1. Framework
- The shared feature extractor serves a labeled few-shot classification branch and an unlabeled self-supervised branch, illustrated with rotation prediction.Task-specific classifiers are trained with their respective few-shot classification and self-supervised losses.Source: §3.1 · p. 3
3.2. Multi-Objective FSAL
- PSST models few-shot auxiliary learning as multi-objective optimization because manually chosen loss weights can be difficult to tune and may not represent task trade-offs adequately.Earlier approaches typically optimize a weighted linear combination of per-task losses.Source: §3.2 · p. 3
3.3.1 Gradient-based multi-objective optimization
- Gradient-based multi-objective optimization balances interactions between shared-parameter task gradients, yielding solutions with balanced performance across tasks.The method updates parameters along a direction satisfying constraints on each task loss.Source: §3.3.1 · pp. 3–4
3.3.2 Optimization in preference region
- PSST decomposes the problem into preference-specific constrained subproblems and excludes regions where auxiliary tasks hamper the main task, reducing the exploration space.It identifies the preference region by comparing main-task performance with the balanced-task performance.Source: §3.3.2 · pp. 4–6
3.4. Preferred Pareto Exploration
- Preferred Pareto exploration uses first-order approximations on tangent planes to explore local Pareto solutions within restricted subregions, then selects the solution best on the main task.The authors attribute lower residual error to this approach; the reported cost of obtaining one tangent direction is O(kn), with k the iteration count and n the input-data size.Source: §3.4 · p. 6
4. Experiments
The experiments test PSST's Pareto exploration and its use in few-shot auxiliary learning. They report state-of-the-art results on MiniImageNet and CIFAR-FS, reduced task conflict, and more efficient exploration.
Experiment design and sufficiency test
- The necessity test compares few-shot methods on MiniImageNet and CIFAR-FS, while the sufficiency test evaluates Pareto exploration using MultiMNIST.In the sufficiency test, the exploration study uses 2048 MultiMNIST images; PSST restricts search by removing regions where auxiliary tasks hamper the main task.Source: §4 · pp. 7–8
Reduce the task conflict
- PSST improves all reported combinations of meta-learning baselines and self-supervised tasks; ProtoNet gains 5.36% in 1-shot and 3.75% in 5-shot with rotation prediction and jigsaw.For BF3S, location prediction lowers performance without PSST, while incorporating PSST improves it by finding a trade-off.Source: §4 · p. 7
Comparison with prior works
- PSST with CC and rotation prediction achieves state-of-the-art results on MiniImageNet and CIFAR-FS.The comparisons use average classification accuracy; Table 2 covers MiniImageNet and Table 3 covers CIFAR-FS.Source: §4 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 8
Training time analysis
- For 10 trade-off solutions, PSST requires 60 × 10^3 iterations versus 249 × 10^3 for BF3S with grid search.For a single solution, the methods' training times are close.Source: §4 · p. 8, Table 4 · p. 8
Decrease residual error
- PSST's exploration trajectories better approximate the Pareto front, which the paper attributes to lower residual error in the restricted subregion.The paper identifies reduced exploration space as a source of efficiency and decreased residual error as a source of effectiveness.Source: §4 · p. 8
Performance comparison of Pareto exploration
- 78 exploration iterations give PSST better main-task performance than previous Pareto exploration, which requires 130 iterations.The methods have the same training time per iteration; Figure 7 compares their performance.Source: §4 · p. 8, Figure 7 · p. 8
5. Conclusion
The conclusion frames task conflict in few-shot auxiliary learning as a multi-objective optimization problem. PSST explores constrained subproblems with different trade-off preferences, and its reported efficiency and effectiveness are attributed to a smaller exploration space and reduced residual error.
Problem and approach
- PSST addresses task conflict in few-shot auxiliary learning by seeking Pareto-optimal network parameters through multi-objective optimization.The conclusion presents the method as an alternative to treating the auxiliary-learning problem as a single weighted objective.Source: §5 · p. 8
- PSST decomposes the auxiliary-learning problem into constrained subproblems with different trade-off preferences to explore the Pareto frontier.The conclusion describes these preferences as enabling better frontier exploration.Source: §5 · p. 8
Conclusion
- The paper attributes PSST's efficiency to less exploration space and its effectiveness to decreasing residual error.These are the stated explanations for the method's reported efficiency and effectiveness.Source: §5 · p. 8