Source-linked AI summary
When is multitask learning effective? Semantic sequence prediction under varying data conditions
Héctor Martínez Alonso, Barbara Plank
TL;DR
The paper asks when multitask learning benefits semantic sequence prediction, an area with limited evidence compared with morphosyntactic tasks. It evaluates auxiliary-task configurations and relates their effects to label-distribution properties, finding that MTL is selective rather than uniformly beneficial and favors compact, relatively uniform distributions.
Problem
Prior work provides limited and mixed evidence about when MTL works for semantic sequence prediction and whether data characteristics predict its success.
Method
The study evaluates semantic main tasks with morphosyntactic and FREQBIN auxiliary tasks, relating performance to label-distribution properties and information-theoretical measures.
Results
MTL is not consistently effective: the strongest auxiliary tasks are FREQBIN or FREQBIN+POS, associated with low kurtosis and fairly high entropy.
Takeaways & Limitations
Successful auxiliary tasks tend to have compact, mid-entropy, low-kurtosis label distributions, while tasks with many labels or a very large O class are less favorable.
Takeaways & Limitations
UNIFORM FREQBIN is not useful in the general case and helps mainly when the main task is already highly co-informative with word frequency.
Abstract
from arXiv · showhide
Multitask learning has been applied successfully to a range of tasks, mostly morphosyntactic. However, little is known on when MTL works and whether there are data characteristics that help to determine its success. In this paper we evaluate a range of semantic sequence labeling tasks in a MTL setup. We examine different auxiliary tasks, amongst which a novel setup, and correlate their impact to data-dependent conditions. Our results show that MTL is not always effective, significant improvements are obtained only for 1 out of 5 tasks. When successful, auxiliary tasks with compact and more uniform label distributions are preferable.
1 Introduction
The paper investigates when multitask learning helps semantic sequence prediction, where prior evidence is limited and mixed. It evaluates auxiliary-task choices against data-dependent label-distribution properties.
- MTL has succeeded across several sequence-prediction tasks, but evidence for more semantic labeling tasks remains limited and mixed.
- Semantic tasks have label distributions that differ from those typical of frequently studied morphosyntactic tasks such as POS tagging.
- The study evaluates MTL for semantic sequence prediction under data-dependent conditions.
- The experiments compare semantic main tasks with morphosyntactic auxiliary tasks, test FREQBIN across tasks, and vary POS auxiliary-task data sources.
- The empirical study finds sensitivity to label-distribution properties and a preference for compact, mid-entropy distributions.
2 Analyzing multi-task learning
The paper frames MTL as shared representation learning across related tasks and analyzes auxiliary tasks using semantic, morphosyntactic, and frequency-based labels. It characterizes datasets through label inventories, O-label proportions, entropy, and kurtosis, while testing several FREQBIN constructions.
- MTL jointly trains tasks with shared hidden layers and task-specific output layers to leverage related-task training signals.
- The main tasks cover semantic sequence labeling, while auxiliary tasks represent frequency and morphosyntactic information through CHUNK, DEPREL, FREQBIN, and POS.
- Dataset descriptions include sentence and token counts, type-token ratio, label-inventory size, and the proportion of O labels.
- The analysis measures label distributions with entropy and kurtosis, capturing uncertainty and distributional tailedness respectively.
- FREQBIN discretizes word log frequency, with SKEWED10, SKEWED5, and UNIFORM providing alternative binning schemes.
- UNIFORM assigns labels using k-quantilized cumulative-frequency indices, while frequency bins are computed from the main-task corpus rather than an external corpus.
3 Model
The model is a shared stacked bi-LSTM that combines word and character representations for multiple sequence-labeling tasks. Task-specific outputs can be attached at different layers, and experiments compare baselines with auxiliary-task combinations.
- The sequence encoder is a bidirectional recurrent model whose representations depend on the full input sequence.
- The architecture combines learned word embeddings with character representations produced by a lower-level bi-LSTM.
- Three stacked bi-LSTMs form shared task layers, with separate softmax output layers for individual tasks.
- Auxiliary tasks can be predicted at inner layers while the main task is predicted at the outermost layer, and training backpropagates sampled task losses through the shared network.
- The experiments use a fixed bi-LSTM setup without task-specific features or pretrained word embeddings to study interactions between main and auxiliary tasks.
- Each main task is evaluated with a no-auxiliary baseline, individual auxiliary systems, and combinations involving the three FREQBIN variants.
4 Results
Across the semantic sequence-labeling experiments, multitask learning produced selective rather than general gains: auxiliary tasks helped some settings, but significant improvement was limited to SEMTRAITS. Results also link MTL behavior to label distributions, auxiliary-task design, training size, and optimization choices.
- Main semantic tasks: Only SEMTRAITS was significantly better than baseline among the two tasks reported as outperforming baseline.Its small label set reduced sparsity when learning shared main–auxiliary label combinations.
- Main semantic tasks: Other low-level tasks did not generally aid prediction of high-level semantic tasks, despite being informative features in single-task learning.The expected benefit from CHUNK to SUPERSENSES did not materialize generally in the MTL setting.
- Auxiliary task contribution: Auxiliary loss failed to benefit bi-LSTM systems when the main task had many labels and excessively high entropy.Among FREQBIN variants, UNIFORM consistently outperformed the skewed base-5 and base-10 versions.
- Auxiliary task contribution: UD/UPOS was the most suitable POS auxiliary source and systematically outperformed UD/PTB and WSJ/PTB alternatives.The authors relate this to sufficient linguistic signal combined with a smaller label inventory.
- FREQBIN: FREQBIN improved POS, SEMTRAITS, and MPQA, indicating benefits beyond lower-level tasks when datasets have suitable data properties.For POS, the X label improved from 75 to 80 and INTJ from 84 to 87, while other gains were at most 1 point.
- Label–frequency co-informativeness: The label–frequency relationship varied substantially across tasks: POS had the highest explanatory power, while FRAMENET’s R2 was slightly negative and rounded to zero.The study estimated R2 by predicting log word frequency from surrounding label trigrams without lexical features.
5 Net capacity and contribution of character representation
Increasing hidden-layer width reduces generalization across all multitask setups, while character features help some tasks but hurt others depending on whether word parts or word identity are informative.
- Net capacity: Increasing hidden-layer size reduces generalization performance across all multitask setups.The expanded network can learn task-specific hidden-layer components, weakening parameter sharing.
- Character representation: MPQA and FRAMES improve by about 2.5 points when character features are disabled, whereas NER drops by up to 8 points.The character-free system relies only on word information.
- Character representation: Character features are uninformative for word-identity-dependent tasks such as FRAMES but useful when word parts indicate labels, as in POS or NER.Character embeddings can approximate capitalization features useful for NER.
- Character representation: Table 5 compares default hierarchical systems using word-plus-character representations with systems using words alone.The comparison isolates the contribution of the lower-level character bi-LSTM.
6 Related Work
Earlier multitask-learning studies largely targeted morphosyntactic or related tasks, while the conditions governing success on semantic sequence tasks remained insufficiently studied. This paper addresses that gap through an extensive evaluation of data properties and task interplay.
- Prior multitask learning: Multitask learning had been applied to chunking, tagging, name error detection, machine translation, and other sequence-prediction tasks.Most earlier work focused on morphosyntactic or related sequence tasks.
- Prior multitask learning: Earlier studies commonly assumed jointly labeled data, whereas this paper evaluates multitask training from distinct datasets.The paper contrasts its setup with prior work using one corpus annotated with multiple labels.
- Open question: Prior results did not establish under what conditions multitask learning works; one earlier study observed improvements only for chunking without examining task data properties.This motivates analyzing data-dependent conditions rather than reporting performance alone.
- Paper contribution: The paper presents the first extensive evaluation, to the authors’ knowledge, of data properties and main-auxiliary task interplay for semantic sequence tasks.Related work also examined auxiliary-data size, including cases where abundant auxiliary data swamped the main task.
7 Conclusions and Future Work
The conclusions link multitask performance to label-distribution properties and task composition, while identifying limits of frequency-based auxiliary tasks and several directions for future work.
- Findings: FREQBIN and FREQBIN+POS are the best auxiliary tasks, combining low kurtosis with fairly high entropy.The study tested three FREQBIN variants and found the novel parametric UNIFORM variant more robust than SKEWED10.
- Findings: UPOS is the most effective evaluated POS auxiliary task, with fewer labels and a more compact, lower-kurtosis distribution than PTB.The comparison concerns three POS data sources differing in corpus composition or label inventory.
- Limitations: UNIFORM is not generally useful as an auxiliary task, helping low-frequency labels mainly when the main task is already highly informative of word frequency.Regression over log frequency would require changing the architecture and joint loss, so the authors leave it for future work.
- Findings: The architecture prefers compact, mid-entropy, low-kurtosis distributions and performs poorly on tasks with many labels or a very large O class.This pattern is presented as a data-property account of semantic-task difficulty rather than a simple morphosyntax-versus-semantics divide.
- Future work: The study lacks Viterbi-style sequence decoding and proposes comparing architectures, additional tasks, loss weighting, and auxiliary labels with predicted features.The authors hypothesize that Viterbi-style decoding could improve prediction of strongly interdependent labels, especially in small datasets or large label inventories.