Source-linked AI summary
Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning
Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, Kyoung Mu Lee
TL;DR
Few-shot learning requires generalization from very few labeled examples, while MAML-based methods often rely on fixed inner-loop losses that may not suit diverse tasks. MeTAL meta-learns a loss function adapted to each task state, and experiments report improved generalization across domains and MAML-based algorithms.
Problem
Few-shot learning must generalize from few labeled examples, but MAML-based methods commonly use fixed losses despite tasks potentially benefiting from different auxiliary or regularization terms.
Method
MeTAL uses meta-learners to learn a loss function and adapt its parameters to each task during inner-loop optimization.
Results
MeTAL improves MAML generalization and consistently improves other MAML-based algorithms across domains and architectures.
Takeaways & Limitations
Learning a task-adaptive inner-loop loss is an important complementary component to improving initialization or weight-update rules.
Takeaways & Limitations
The task formulation assumes each task is drawn from a task distribution and consists of disjoint support and query sets.
Abstract
from arXiv · showhide
In few-shot learning scenarios, the challenge is to generalize and perform well on new unseen examples when only very few labeled examples are available for each task. Model-agnostic meta-learning (MAML) has gained the popularity as one of the representative few-shot learning methods for its flexibility and applicability to diverse problems. However, MAML and its variants often resort to a simple loss function without any auxiliary loss function or regularization terms that can help achieve better generalization. The problem lies in that each application and task may require different auxiliary loss function, especially when tasks are diverse and distinct. Instead of attempting to hand-design an auxiliary loss function for each application and task, we introduce a new meta-learning framework with a loss function that adapts to each task. Our proposed framework, named Meta-Learning with Task-Adaptive Loss Function (MeTAL), demonstrates the effectiveness and the flexibility across various domains, such as few-shot classification and few-shot regression.
1. Introduction
Few-shot learning must generalize from very few labeled examples, but MAML-based methods commonly retain fixed inner-loop losses that may be inadequate for diverse tasks. MeTAL addresses this by meta-learning a task-adaptive loss function and improves generalization across domains and MAML-based algorithms.
- Few-shot learning seeks generalization to unseen examples from only a few labeled support examples, while avoiding overfitting.
- MAML learns an initialization for rapid adaptation, but diverse training and test tasks can still make generalization difficult.
- MAML variants often use simple fixed losses even though auxiliary terms such as ℓ2 regularization can improve generalization.
- MeTAL meta-learns a loss function whose parameters adapt to each task during inner-loop optimization.It uses one meta-learner to learn the loss function and another to generate task-dependent parameter transformations.
- MeTAL improves MAML generalization and remains effective across different domains and other MAML-based algorithms.The framework can combine labeled support and unlabeled query examples during task adaptation.
2. Related Work
Few-shot meta-learning transfers prior knowledge to new tasks through several learning paradigms, including optimization-based methods such as MAML. Related approaches improve initialization or update rules, whereas MeTAL makes the inner-loop loss itself task-adaptive and can complement these methods.
- Few-shot learning aims to adapt to new tasks from few examples while generalizing to unseen examples without overfitting.
- Meta-learning learns prior knowledge from previous tasks and uses it to adapt rapidly to new tasks.
- Optimization-based approaches formulate meta-learning as bi-level optimization, with adaptation in an inner loop and generalization training in an outer loop.
- MAML represents prior knowledge as a learnable initialization for gradient-based fine-tuning across diverse problem domains.
- Earlier methods primarily improve initialization or inner-loop updates while retaining a common task loss, despite the potential value of auxiliary regularization.
- MeTAL introduces a task-adaptive loss function that preserves simplicity and can be used jointly with other MAML-based methods.
3. Proposed Method
MeTAL extends optimization-based meta-learning by adapting both the loss function and its parameters to each task’s intermediate state. The resulting inner-loop procedure can incorporate labeled support and unlabeled query information while retaining a gradient-based base learner update.
- Problem formulation: Meta-learning samples tasks from a task distribution and adapts a base learner using support and query sets.
- Optimization: MeTAL performs gradient-based adaptation with the task-adaptive loss and updates θ, ϕ, and ψ through outer-loop optimization.
- MAML: In standard MAML, inner-loop gradient descent adapts base-learner weights from a shared initialization using the support-set loss.
- MAML: MAML evaluates the adapted learner on unseen query examples and uses that generalization signal to update the initialization in the outer loop.
- Task-adaptive loss function: MeTAL replaces the fixed inner-loop objective with a meta-learned loss function whose parameters are adapted according to the current task state.
- Task-adaptive loss function: A meta-network generates affine parameters γ and β that transform the loss parameters as ϕ′ = γϕ + β.
- Task state: The task state can include support losses, base-learner weights, outputs, and responses on unlabeled query examples.
- Architecture: The task state is kept computationally minimal, with dimension 1 + L + N for an L-layer base learner producing N-dimensional outputs.
4. Experiments
Experiments evaluate MeTAL across few-shot classification, cross-domain classification, regression, and ablations. Across these settings, task-adaptive learned losses consistently improve generalization and complement existing MAML variants and adaptation strategies.
- Few-shot classification: MeTAL improves MAML and remains compatible with MAML++ and ALFA on miniImageNet and tieredImageNet classification.The authors describe the improvement as substantial and further improvement when MeTAL is combined with other MAML variants.
- Cross-domain classification: MeTAL improves cross-domain performance by approximately 8%, compared with approximately 4% in standard few-shot classification.Models are trained on miniImageNet and evaluated on CUB, making this a domain-shift setting.
- Cross-domain classification: MeTAL’s improvement remains as large when combined with ALFA as when combined with MAML, supporting complementary benefits from adaptive losses and weight-update rules.The comparison distinguishes learning the inner-loop loss function from learning a new gradient-based update rule.
- Few-shot regression: MeTAL consistently improves MAML under 5-shot, 10-shot, and 20-shot regression settings measured by mean-square error.The regression tasks use sinusoidal target functions with varying amplitude, frequency, and phase.
- Ablation studies: Ablations show that learned losses improve MAML, task-adaptive losses outperform fixed learned losses, and semi-supervised optimization alone remains below full MeTAL.The results also indicate that conditioning the task state on cross-entropy loss supports better generalization, while omitting it yields an unsupervised inner loop that struggles.
- Ablation studies: Generated affine parameters vary across inner-loop steps and tasks, indicating that MeTAL dynamically adapts the loss during task optimization.Variation is especially pronounced among tasks at the last inner-loop step.
5. Conclusion
MeTAL learns a loss function that adapts to each task during inner-loop optimization, supporting better generalization across MAML variants and problem domains. Its experiments emphasize the importance of task-specific loss learning in few-shot learning.
- MeTAL learns a task-adaptive loss function from the current task state during inner-loop optimization.The framework is designed to learn the loss specifically needed by each task for better generalization.
- MeTAL supports semi-supervised inner-loop optimization by jointly using labeled support and unlabeled query examples.
- MeTAL can be applied across different MAML variants and problem domains.
- The experiments underline the importance of learning a good loss function alongside weight-update rules or initialization.
A. Loss Landscape Visualization
The loss-landscape analysis compares MAML and MeTAL using loss variation, gradient predictiveness, and effective β-smoothness. These measures characterize optimization stability and smoothness in the inner loop.
- The visualization analyzes inner-loop optimization landscapes by measuring loss variations, gradient predictiveness, and effective β-smoothness.Gradient predictiveness concerns changes in gradients, while effective β-smoothness measures maximum gradient differences.
- Figure A presents the inner-loop loss landscape visualization for comparing MAML and MeTAL.
B.1. CIFAR100-based Datasets
On CIFAR-FS and FC100, MeTAL achieves state-of-the-art performance with comparable base-learner backbones and improves consistently across baselines, backbones, and datasets. It also outperforms methods using pretrained or larger networks, including transductive competitors.
- Results: MeTAL achieves state-of-the-art performance with ALFA using the same base-learner backbone on CIFAR-FS and FC100.The comparison uses 4-CONV or ResNet12 backbones.
- Results: MeTAL outperforms recent methods using pretrained networks or the larger WRN-28-10 network, especially in 5-shot classification.
- Results: MeTAL outperforms SIB and its variants while using the same or sometimes smaller backbone.SIB and SIB + E3BM explicitly use a transductive setting, like MeTAL.
- Results: MeTAL provides consistent performance improvement across different baselines, base-learner backbones, and datasets.
- Evaluation: Table A reports 5-way 1-shot and 5-way 5-shot classification test accuracy on CIFAR-FS and FC100.
B.2. Detailed comparisons on ImageNet-based Datasets
MeTAL outperforms most recent optimization-based meta-learners with similar base-learner backbones, including methods that use pretrained feature extractors. It achieves competitive performance without relying on pretraining.
- MeTAL outperforms most recent optimization-based meta-learners with similar 4-CONV or ResNet12 base-learner backbones.
- MeTAL remains competitive against methods using pretrained feature extractors.The passage notes that pretrained extractors may limit application or effectiveness to classification problems.
C. Semi-Supervised Inner-Loop Optimization
MeTAL uses unlabeled examples during semi-supervised inner-loop optimization and extracts task-relevant information rather than simply overfitting to the given query set. Its accuracy improves with more non-query examples and remains relatively stable as distractor examples increase.
- Evaluation: The evaluation reports 5-way 1-shot and 5-way 5-shot test accuracy on miniImageNet and tieredImageNet.The corresponding comparison includes multiple MAML-based models and baselines.
- Mechanism: MeTAL extracts information from unlabeled images instead of merely fine-tuning to the given query set.Ablation evidence indicates that the method does not learn to fine-tune to the query set but learns to use information from unlabeled images.
- Setting: The semi-supervised setting divides unlabeled data into query, non-query, and distractor sets based on task membership and inference timing.Non-query examples share classes with the query set and are available before inference, whereas distractors come from different tasks.
- Results: Accuracy increases with the number of non-query unlabeled examples, indicating improved generalization from relevant unlabeled information.The comparison uses 5-way 5-shot miniImageNet classification with a 4-CONV backbone.
- Results: MeTAL's accuracy does not drop significantly as the number of distractor sets increases.The reported result suggests some robustness to irrelevant or potentially destructive unlabeled examples.
D. Visual Tracking
MeTAL is applied to MetaCREST for visual tracking, where the target is specified by a bounding box in the first video frame. Quantitative and qualitative results show improved performance over MetaCREST on OTB2015 sequences.
- Task and method: Visual tracking is treated as a few-shot problem because only the target bounding box in the first video frame is given.MeTAL is applied to the MAML-based MetaCREST tracker to handle object appearance changes across video frames.
- Results: MeTAL improves MetaCREST performance quantitatively and qualitatively in visual tracking.The evaluation uses results presented in Table D and Figure B.
- Qualitative results: Figure B shows MetaCREST and MetaCREST+MeTAL bounding boxes across selected OTB2015 frames.Yellow denotes MetaCREST, red denotes MetaCREST+MeTAL, and rows correspond to bird1, diving, dragonBaby, and girl2.
- Adapter behavior: The adapter meta-network exhibits dynamic behavior across inner-loop steps and tasks, including changing the second-layer bias offset while minimizing scaling.This behavior is illustrated through affine transformation parameters generated for loss-learner layers.
F. Implementation Details
The implementation evaluates standard few-shot classification settings with 4-CONV and ResNet12 backbones, repeated across selected models and random seeds. MeTAL's loss adapter is a two-layer MLP that generates affine transformations for the loss learner from task-state information.
- Experimental setup: Experiments use 4-CONV and ResNet12 base learners with fixed few-shot evaluation settings.Each task has 15 query examples per class, and reported results use ensembles of five top-validation models repeated with three random seeds.
- Backbones: The 4-CONV backbone has four convolution layers, while ResNet12 has four residual blocks.Both architectures end with a fully connected layer for classification; the 4-CONV design also includes batch normalization, Leaky ReLU, and max pooling.
- Loss adapter: The loss adapter gψ is a two-layer MLP with ReLU activation between layers.Its input includes a classical loss value, layer-wise mean weights, and the base learner output.
- Semi-supervised input: For unlabeled examples, output entropy replaces cross-entropy because ground-truth labels are unavailable.The adapter input dimension remains matched to the semi-supervised setting.
- Parameter generation: The adapter outputs affine parameters γ and β for the weights and biases of each loss-learner layer.For a two-layer loss learner, the output dimension is 4Lϕ = 8.