Source-linked AI summary
Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network
Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, Ting Liu
TL;DR
Few-shot slot tagging must handle scarce labels while modeling dependencies across domain-specific label sets. The paper addresses this with collapsed dependency transfer in a CRF and L-TapNet, which uses label-name semantics for emissions. Experiments report significant improvements over strong few-shot baselines, with the abstract reporting a 14.64 F1-score one-shot gain.
Problem
Few-shot slot tagging has scarce labeled data and cannot directly transfer learned label dependencies when source and target domains use different label sets.
Method
The paper combines a few-shot CRF with collapsed dependency transfer for transition scores and L-TapNet for word-label similarity using label-name semantics.
Results
The model significantly improves over strong few-shot baselines in one-shot and five-shot experiments, and both proposed components improve tagging accuracy.
Takeaways & Limitations
Abstracting labels enables dependency transfer across domains with different label sets, while label-name semantics enhance label representations for slot tagging.
Takeaways & Limitations
The formulation assumes the same number of labels across domains and includes Start and End as additional abstract labels in practice.
Abstract
from arXiv · showhide
In this paper, we explore the slot tagging with only a few labeled support sentences (a.k.a. few-shot). Few-shot slot tagging faces a unique challenge compared to the other few-shot classification problems as it calls for modeling the dependencies between labels. But it is hard to apply previously learned label dependencies to an unseen domain, due to the discrepancy of label sets. To tackle this, we introduce a collapsed dependency transfer mechanism into the conditional random field (CRF) to transfer abstract label dependency patterns as transition scores. In the few-shot setting, the emission score of CRF can be calculated as a word's similarity to the representation of each label. To calculate such similarity, we propose a Label-enhanced Task-Adaptive Projection Network (L-TapNet) based on the state-of-the-art few-shot classification model -- TapNet, by leveraging label name semantics in representing labels. Experimental results show that our model significantly outperforms the strongest few-shot learning baseline by 14.64 F1 scores in the one-shot setting.
1 Introduction
Few-shot slot tagging must model label dependencies despite scarce labels and changing domains. The paper combines collapsed dependency transfer in a CRF with L-TapNet, which uses label-name semantics for word-label similarity.
- Few-shot slot tagging adapts to new domains from only one or two labeled examples per class, addressing scarce labels in rapidly changing domains.
- Sequence labeling requires modeling label dependencies, so the framework uses a CRF with similarity-based emissions and transferred transition scores.
- Collapsed dependency transfer abstracts domain-specific labels into domain-independent labels and models dependencies between them for cross-domain transfer.
- L-TapNet leverages label-name semantics to construct separated, aligned label representations for computing word-label similarity.
- Ablation tests attribute improvements to both L-TapNet and collapsed dependency transfer, while dependency analysis finds gains over rule-based transitions.
2 Problem Definition
The paper defines few-shot sequence labeling as selecting the best label sequence for a query given a small support set. The setup trains on source domains and evaluates on unseen target domains without fine-tuning.
- A sentence is represented as a word sequence x and its labels as a corresponding sequence y.
- Each domain has a domain-specific label set, with the formulation assuming the same number of labels across domains.
- Few-shot models train on source domains, then operate on unseen target domains using only a small labeled support set.
- Given a K-shot support set and query sequence x, the task is to find the query’s best label sequence y∗.
3 Model
The model combines a CRF transition scorer with an emission scorer for few-shot slot tagging. Collapsed dependency transfer handles label-set mismatch across domains, while L-TapNet uses task-adaptive projections and label semantics to compute word-label similarity.
- 3.1 Framework Overview: The framework applies a linear-CRF to predict label sequences from a query sentence and a K-shot support set.Its two components are a Transition Scorer and an Emission Scorer.
- 3.2 Transition Scorer: A trained transition matrix cannot directly transfer to a target domain when source and target domains use different label sets.This label mismatch blocks direct application of the source-domain transition scorer.
- 3.2 Transition Scorer: Collapsed dependency transfer abstracts specific labels into O, B, and I, modeling transitions among same- or different-type B and I labels.The abstract transitions are stored in a 3×5 table ˜T and used to construct domain-specific transition matrices.
- 3.3 Emission Scorer: L-TapNet computes emission scores from word-label similarity in a projected embedding space where different labels are separated.It extends TapNet with label semantics and prototypes for label representation.
- 3.3.1 Task-Adaptive Projection Network: The projection M aligns support-set prototypes with references while reducing alignment error and separating words from different labels.TapNet constructs M through a linear error nulling process.
- 3.3.2 Label-enhanced TapNet: L-TapNet combines domain-agnostic references with label-name semantics and prototypes to form label representations for emission scoring.The label-enhanced reference uses ψj = (1 − α) · φj + αsj, while the final representation combines cj and ψj.
4 Experiment
The experiments evaluate few-shot slot tagging across domains using sentence-based support sets and cross-domain episodes. The proposed components improve performance, with gains attributed to dependency transfer, label-enhanced projections, and domain-adaptive representations.
- Settings: Few-shot episodes pair a query utterance with a K-shot support set, evaluated across target domains while training uses the remaining source domains.The evaluation trains on 10,000 samples and validates and tests on 2,000 samples respectively.
- Few-shot Data Construction: Support sentences are sampled so every domain label appears at least K times, although co-occurring labels make exact K-shot construction impossible.The procedure approximately constructs K-shot sets and may randomly skip removals with 20% probability.
- Main Results: 14.64 average F1 separates L-TapNet+CDT from the strongest few-shot baseline WPZ+BERT in the 1-shot setting.The model achieves the best performance among the reported methods.
- Main Results: The model significantly outperforms Bi-LSTM and TransferBERT, while metric-based methods produce more reasonable predictions than conventional fine-tuned models.Fine-tuning on extremely limited examples leads to poor generalization and an undertrained classifier.
- Main Results: L-TapNet+CDT improves over L-WPZ+CDT by 4.79 F1 and over TapNet+CDT by 4.54 average F-score.These comparisons support benefits from embedding projection, label semantics, and prototype-based emission representations.
- Ablation Analysis: Removing collapsed dependency transfer causes large F1 drops, while removing label semantics, pair-wise embeddings, or prototype references also reduces performance.The ablations attribute gains to label dependencies, label-name representations, domain context, and support-derived prototypes.
- Dependency Analysis: Collapsed dependency transfer improves inner and border label-bigram accuracy, helping reduce illegal transitions and identify slot boundaries.The analysis reports improvements for both within-span transitions and the first and last tokens of slot spans.
5 Related Works
Prior few-shot NLP work largely addresses classification, while few-shot slot tagging remains less investigated. Existing approaches include metric learning, class-name semantics, transition rules, and task-specific additions.
- Few-shot Learning: Few-shot classification commonly predicts queries by similarity to class representations built from support examples.These methods learn similarity functions in rich-resource domains and form per-class representations from few labeled samples.
- Few-shot Learning: Recent few-shot methods use class-name semantics to enhance class representations, but the cited approaches focus on image classification rather than slot tagging.The paper positions its label-semantic approach within this broader line of work.
- Transition Modeling: A transition rule baseline greedily predicts each word’s label and blocks results that conflict with the previous label.The paper compares this rule-based strategy with collapsed dependency transfer in its label-dependency analysis.
- Few-shot NLP: Few-shot NLP research has covered text classification, entity relation classification, and dialog act prediction, whereas few-shot slot tagging is less investigated.Prior slot-tagging work used additional regular expressions, making it incomparable to the paper’s model because of that extra information.
6 Conclusion
The paper proposes a few-shot CRF for task-oriented dialogue slot tagging, combining collapsed dependency transfer for transitions with L-TapNet for emissions. Experiments validate that both mechanisms improve tagging accuracy.
- 6 Conclusion: The model transfers label dependencies across domains with different label sets through collapsed dependency transfer.This mechanism estimates CRF transition scores under few-shot conditions.
- 6 Conclusion: L-TapNet improves CRF emission-score calculation by enhancing label representations with label-name semantics.Emission scores are computed from word-label similarity.
- 6 Conclusion: Both collapsed dependency transfer and L-TapNet improve tagging accuracy.
A Detail of Dataset
The dataset section identifies the original dataset statistics used to construct the few-shot experiment data.
- A Detail of Dataset: Table 7 reports statistics for the original dataset used to construct the few-shot experiment data.
- A Detail of Dataset: The few-shot experiment data are constructed from the original dataset.
- A Detail of Dataset: The original-dataset statistics are presented in a table titled “Statistic of Original Dataset.”
B Few-shot experiments for Name entity recognition
Few-shot NER experiments evaluate the model across four domains and show stronger performance for the proposed model, with domain gap and shot-dependent transfer effects shaping results.
- B Few-shot experiments for Name entity recognition: Few-shot NER experiments use CoNLL-2003, GUM, WNUT-2017, and Ontonotes from News, Wiki, Social, and Mixed domains.Each dataset contains data from one domain.
- B Few-shot experiments for Name entity recognition: The best proposed model outperforms all baselines in both one-shot and five-shot NER settings.Tables 9 and 10 report the corresponding results.
- B Few-shot experiments for Name entity recognition: NER scores are lower than slot-tagging scores because the NER domains come from different datasets and have a larger domain gap.
- B Few-shot experiments for Name entity recognition: The improvement margin narrows in the five-shot setting because different NER genres and vocabularies make domain-specific support examples more beneficial.
- B Few-shot experiments for Name entity recognition: Ablating collapsed dependency transfer and label semantics causes performance drops in both one-shot and five-shot settings.The ablation results are averaged F1 scores across domains.
C Analysis of Projection Space Dimensionality
Increasing L-TapNet’s projected-space dimensionality improves one-shot SNIPS performance until the gains gradually stabilize, indicating that lower dimensions may remain viable.
- C Analysis of Projection Space Dimensionality: One-shot SNIPS performance improves as L-TapNet’s projected-space dimension increases, then gradually stabilizes.The analysis examines different dimensions of the mapping space.
- C Analysis of Projection Space Dimensionality: The dimensionality trend suggests that projection-space reduction may retain most performance.The passage describes this as a possibility rather than a demonstrated optimal dimension.
D Slot Tagging Result with Standard Deviations
Tables 12 and 13 report complete slot-tagging results with standard deviations for the 1-shot and 5-shot settings.
- Together, Tables 12 and 13 provide the complete slot-tagging results with standard deviations.
- Table 12 reports 1-shot slot-tagging results with standard deviations.
- Table 13 reports 5-shot slot-tagging results with standard deviations.