Source-linked AI summary
Curriculum Learning of Multiple Tasks
Anastasia Pentina, Viktoriia Sharmanska, Christoph H. Lampert
TL;DR
Multi-task learning benefits from shared information, but not all tasks are equally related, raising the question of how transfer should be organized. The paper learns tasks sequentially, uses a generalization bound to choose their order, and finds that sequential learning can outperform joint learning while automatically discovering advantageous sequences.
Problem
Multi-task learning lacks an equally suitable sharing strategy when tasks differ in relatedness, even though shared information can improve generalization with limited labeled data.
Method
The paper decomposes multi-task learning into sequential domain-adaptation steps, transferring information between subsequent tasks and selecting their order using a generalization-bound criterion.
Results
Sequential learning can be more effective than joint or independent learning, task order affects overall classification performance, and the algorithm automatically discovers beneficial orders.
Takeaways & Limitations
Task curriculum and selective sequential transfer provide an alternative to symmetric joint learning when tasks are not equally related.
Takeaways & Limitations
The model transfers only from the previous task, producing sequences or multiple subsequences rather than tree- or graph-structured task organizations.
Abstract
from arXiv · showhide
Sharing information between multiple tasks enables algorithms to achieve good generalization performance even from small amounts of training data. However, in a realistic scenario of multi-task learning not all tasks are equally related to each other, hence it could be advantageous to transfer information only between the most related tasks. In this work we propose an approach that processes multiple tasks in a sequence with sharing between subsequent tasks instead of solving all tasks jointly. Subsequently, we address the question of curriculum learning of tasks, i.e. finding the best order of tasks to be learned. Our approach is based on a generalization bound criterion for choosing the task order that optimizes the average expected classification performance over all tasks. Our experimental results show that learning multiple related tasks sequentially can be more effective than learning them jointly, the order in which tasks are being solved affects the overall performance, and that our model is able to automatically discover the favourable order of tasks.
1. Introduction
Multi-task learning can improve generalization with less labeled data, but treating all tasks symmetrically may be suboptimal when task relationships differ. The paper therefore proposes sequential transfer and a theoretically guided curriculum for ordering tasks.
- Sharing information between related tasks can improve generalization from fewer training points per task than learning each task independently.Large labeled datasets are expensive and time-consuming to obtain, particularly in computer vision.
- Prototype-based multi-task learning treats tasks symmetrically, which may be unsuitable when some tasks or task groups are unrelated.The paper measures task similarity through Euclidean distance between corresponding weight vectors.
- The proposed approach transfers information sequentially from a previously learned task to the next instead of solving all tasks simultaneously.This design is intended to accommodate greater task variability and avoid processing all training data at once.
- PAC-Bayesian analysis yields a generalization bound that evaluates task orders and supports an algorithm for automatically selecting a favorable sequence.Experiments on two real-world image datasets compare sequential learning with independent and jointly solved tasks.
- Sequential learning can outperform both independent learning and standard joint multi-task learning, while the chosen task order affects overall performance.The experiments also report that the proposed algorithm reliably discovers an advantageous order.
2. Related Work
Related work develops multi-task transfer through latent representations, graph or correlation-based regularization, and curriculum ordering of training examples. These approaches motivate the paper while exposing computational or prior-knowledge requirements.
- Latent-basis methods represent related-task parameters as sparse linear combinations of common basis vectors and can allow partial overlap between task groups.The approach was also adapted to lifelong learning.
- Subspace-based methods become computationally costly as feature dimensionality increases, limiting their applicability to the targeted computer vision problems.An exception required dimensionality reduction when jointly learning multiple attribute predictors.
- Graph regularization and correlation-based penalties relax the assumption that all tasks are related, but require prior knowledge about task similarities.The paper contrasts this requirement with its own ordering procedure.
- Earlier curriculum-learning work mainly orders training examples by difficulty, whereas this paper studies ordering multiple learning tasks.Prior studies reported faster training and higher prediction quality from gradually increasing example difficulty.
3. Method
The method decomposes multi-task learning into sequential domain-adaptation problems, transferring information between subsequent tasks. It derives a bound for evaluating task orders and uses it to choose sequences or multiple subsequences when tasks are not uniformly related.
- Sequential learning: The learner processes tasks sequentially, transferring the previous task’s weight vector to the next instead of solving all tasks jointly.The first task uses a standard linear SVM, while subsequent tasks use Adaptive SVM with the previous solution as the reference vector.
- Sequential learning: The approach does not require all tasks to be equally related, but its performance depends on consecutive tasks being related.Outlier tasks or groups of unrelated tasks motivate forming separate subsequences rather than one sequence.
- Data-dependent ordering: The generalization-bound criterion combines empirical prediction confidence with the L2 similarity between weight vectors of subsequent tasks.The resulting bound depends on the task order and provides a computable quality measure for selecting it.
- Data-dependent ordering: Because exhaustive search over all permutations is expensive, SeqMT incrementally selects the unused task minimizing the bound-related objective.At each step, the selected task is both easy according to empirical error and similar to the preceding task in weight space.
- Multiple subsequences: MultiSeqMT extends SeqMT by allowing the learner to continue an existing subsequence or start a new one, transferring information only within subsequences.For each subsequence, it first identifies the most promising next task, then chooses the subsequence-task pair with the minimal criterion value.
4. Experiments
The experiments test whether sequential transfer and learned task orders improve multi-task classification when tasks are not equally related. Across AwA and Shoes, SeqMT and MultiSeqMT generally outperform joint, independent, fixed-order, and random-order alternatives.
- Learning task orders: On AwA, SeqMT outperforms the joint MT and independent IndSVM baselines in all eight cases.The comparison supports sequential transfer when the tasks are not equally related.
- Learning task orders: SeqMT learns an order that is better than or on par with the human easy-to-hard Semantic order in six of eight AwA classes.It fails to match Semantic for chimpanzee and giant panda, while outperforming the best fixed orders for rat and seal.
- Learning task orders: The adaptive SeqMT order is highly competitive with all possible fixed orders and is advantageous over solving tasks in a fixed order.The violin plot represents the performance distribution of all possible orders, while the learned order varies across repeats.
- Order-selection objective: The complexity term measuring task similarity is more important than training error for selecting the next task, although combining both terms is never worse and sometimes better.The complexity term is based on similarity between tasks, while the combined objective retains the error term.
- Learning related subsequences: On Shoes, MultiSeqMT is the most effective strategy, while SeqMT performs on par with RandomMultiSeq and better than Random.The results indicate that multiple related subsequences can avoid transfer between unrelated tasks; related attributes frequently form recurring subsequences.
5. Conclusion
The paper concludes that sequential multi-task learning can outperform joint learning, task order affects classification performance, and the method can discover beneficial orders. Its current limitation is transferring only from the previous task, producing sequences or subsequences.
- Sequential learning can be more effective than joint learning, while task order affects overall classification performance.
- The proposed method automatically discovers a beneficial order of tasks.
- The model transfers only from the previous task, so it produces sequences of related tasks or multiple task subsequences.
A. Proof of Theorem 1
The proof applies PAC-Bayesian analysis to sequentially learned tasks, where each task posterior can serve as prior knowledge for the next. It bounds the difference between expected and empirical average error under fixed or uniformly selected task orders.
- PAC-Bayesian theory provides a generalization bound for sequential task solving.
- Each task posterior distribution is learned from its training set and prior knowledge, then used as prior information for the next task.
- Because task distributions are unknown, expected error is approximated using empirical error on observed training data.
- Theorem 2 bounds the difference between expected and empirical quantities with probability at least 1 −δ.
- A union-bound extension makes the generalization result hold uniformly over all task orders.
B. Additional information for MultiSeqMT
MultiSeqMT extends sequential learning by allowing information transfer to stop between unrelated tasks, thereby forming multiple subsequences. Its bounds and algorithm handle task orders and transfer flags uniformly.
- The extended model can use the fixed original distribution instead of the previous task posterior when transfer is not informative.
- Theorem 4 gives a bound for sequential learning with fixed transfer flags indicating whether adjacent tasks share information.
- A union-bound argument extends the result uniformly across task orders and all possible subsequence partitions.
- MultiSeqMT iteratively chooses unused tasks while deciding whether to continue an existing subsequence or start a new one.
C. Additional information for experiments
The supplementary experiment information describes tables for the AwA and Shoes datasets. Table 3 reports error estimates for easy-to-hard task sequences, while Table 4 records class encodings for shoe attributes.
- Table 3 reports average error rate and standard error over 20 repeats for each AwA class and method.
- Table 4 records class orderings for Shoes attributes, with blue cells marking negative examples and yellow cells marking positive examples.