Source-linked AI summary
DIET: Lightweight Language Understanding for Dialogue Systems
Tanja Bunk, Daksh Varshneya, Vladimir Vlasov, Alan Nichol
TL;DR
The paper asks whether dialogue NLU needs large pre-trained language models and introduces DIET, a flexible multi-task architecture for intent and entity prediction. DIET combines sparse and optional dense representations, achieves strong results without pre-training, and its best configuration outperforms fine-tuned BERT while training six times faster.
Problem
Large pre-trained language models perform strongly, but their computational cost is considerable; dialogue applications therefore need evidence about fast training and performance without large-scale pre-training.
Method
DIET is a multi-task architecture for intent classification and entity recognition that combines sparse word and character n-gram features with optional pre-trained embeddings.
Results
DIET advances state of the art on the challenging NLU-Benchmark dataset; its best pre-trained-embedding configuration outperforms fine-tuning BERT and is six times faster to train.
Takeaways & Limitations
Pre-trained embeddings are not required for competitive DIET performance, and no single embedding set is best across datasets, supporting a modular architecture.
Takeaways & Limitations
The paper hypothesizes, rather than establishes, that masked-input reconstruction regularizes training and promotes more general text features.
Abstract
from arXiv · showhide
Large-scale pre-trained language models have shown impressive results on language understanding benchmarks like GLUE and SuperGLUE, improving considerably over other pre-training methods like distributed representations (GloVe) and purely supervised approaches. We introduce the Dual Intent and Entity Transformer (DIET) architecture, and study the effectiveness of different pre-trained representations on intent and entity prediction, two common dialogue language understanding tasks. DIET advances the state of the art on a complex multi-domain NLU dataset and achieves similarly high performance on other simpler datasets. Surprisingly, we show that there is no clear benefit to using large pre-trained models for this task, and in fact DIET improves upon the current state of the art even in a purely supervised setup without any pre-trained embeddings. Our best performing model outperforms fine-tuning BERT and is about six times faster to train.
1 Introduction
Dialogue NLU commonly separates intent classification and entity recognition, but joint modeling can reduce error propagation. DIET combines these tasks while targeting fast iteration and evaluating performance with and without large-scale pre-training.
- Dialogue NLU typically comprises intent classification and entity recognition.
- Separate modeling of intent and entity tasks can suffer from error propagation, motivating a shared multi-task architecture.
- Large pre-trained language models perform strongly on language-understanding benchmarks but impose considerable pre-training and fine-tuning costs.
- Dialogue assistants need models that can be trained and iterated quickly, including in languages without large-scale pre-training resources.
- DIET jointly models intent classification and entity recognition, supports plug-and-play pre-trained embeddings with sparse features, and improves state of the art without pre-trained embeddings.
- The best DIET model significantly outperforms fine-tuned BERT and trains six times faster.
2 Related Work
Prior work uses dense representations from large pre-trained language models and develops joint architectures for intent classification and named entity recognition. These lines of work motivate DIET’s study of representations and multi-task modeling.
- Large pre-trained language models provide transferable word and sentence representations that have benefited language-understanding benchmarks.
- Earlier multi-task approaches jointly predict intent and named entities using recurrent architectures, including BiGRU and attention-based BiLSTM models.
3 DIET Architecture
DIET combines sparse token and character n-gram features or dense pre-trained embeddings with a transformer encoder for joint intent and entity modeling. It predicts entities with a CRF, intents through embedding-space similarity, and can add masked-token training alongside the task losses.
- Featurization: DIET featurizes token sequences with sparse word and character n-gram features, optionally combined with dense pre-trained embeddings.Character n-grams use n ≤5 and sparse features receive dropout to reduce overfitting.
- Transformer: The architecture uses a two-layer transformer with relative-position attention to encode context across the complete sentence.
- Transformer: DIET applies shared feed-forward weights across sequence steps before the transformer dimension is matched.
- Named entity recognition: Entity labels are predicted with a CRF tagging layer on top of the transformer output sequence.
- Intent classification: Intent classification embeds the transformer’s classification token and intent labels in one semantic space and uses dot-product similarity for prediction.At inference, dot-product similarity ranks all possible intent labels.
- Masking: Masked-token training selects 15% of input tokens and adds a dot-product loss for reconstructing their labels.Selected tokens are masked in 70% of cases, randomized in 10%, and left unchanged in 20%.
- Total loss: DIET minimizes intent, entity, and masking losses jointly, while allowing any loss to be disabled.
4 Experimental Evaluation
DIET is evaluated on three dialogue NLU datasets, with NLU-Benchmark as the primary and most challenging setting. Experiments compare feature combinations, joint versus single-task training, fine-tuned BERT, and transfer to ATIS and SNIPS.
- Datasets: The evaluation uses NLU-Benchmark, ATIS, and SNIPS, focusing on NLU-Benchmark because it is the most challenging dataset.ATIS and SNIPS already have state-of-the-art test accuracy close to 100%.
- Evaluation setup: NLU-Benchmark evaluation averages micro-averaged precision, recall, and F1 across ten independently trained folds.An entity is counted as a true positive when its predicted span overlaps the gold span and the labels match.
- NLU-Benchmark results: DIET’s best NLU-Benchmark model combines token and character n-gram sparse features with ConveRT embeddings and exceeds HERMIT by over 2% on intents.Its entity micro-averaged F1 is 86.04%, versus 84.74% for HERMIT; recall is 86.13% versus 82.04%.
- Ablation study: Joint training slightly lowers intent classification performance compared with single-task training, from 90.90% to 90.18%.The experiment tests whether intent classification and entity recognition benefit from shared optimization.
- BERT comparison: Fine-tuned BERT is outperformed for entity recognition and matched for intent classification, while taking 60 hours versus 10 hours for DIET with ConveRT and sparse features.The comparison uses frozen ConveRT embeddings together with word- and character-level sparse features.
- Transfer evaluation: DIET with sparse features alone reaches performance within 1-2% of Joint BERT on ATIS and SNIPS, while using transferred hyperparameters.This evaluates whether configurations selected on NLU-Benchmark transfer across datasets.
5 Conclusion
DIET is a flexible architecture for joint intent and entity modeling that advances state of the art on the challenging NLU-Benchmark dataset. Its results show that embedding choices matter across datasets, while sparse features alone remain competitive and the best DIET configuration exceeds fine-tuned BERT with lower training time.
- DIET provides a flexible architecture for jointly modeling intent classification and entity recognition.
- DIET advances the state of the art on the challenging NLU-Benchmark dataset.
- No single embedding set performs best across datasets, highlighting the importance of a modular architecture.
- 60 hours versus 10 hours: fine-tuning BERT inside DIET takes six times longer than DIET with sparse and ConveRT features.
- Sparse word and character n-gram features alone enable DIET to outperform the state of the art on NLU-Benchmark.