Source-linked AI summary
Discovering New Intents with Deep Aligned Clustering
Hanlei Zhang, Hua Xu, Ting-En Lin, Rui Lyu
TL;DR
Discovering new intents is difficult when methods have limited ability to transfer known-intent knowledge and provide high-quality clustering supervision. Deep Aligned Clustering uses limited labeled examples and centroid alignment to create pseudo-label supervision; it reports better results across two benchmark datasets.
Problem
Existing methods are limited in transferring prior knowledge from known intents and constructing high-quality supervision for clustering unlabeled known and new intents.
Source: Introduction · p. 1Method
Deep Aligned Clustering uses k-means assignments as pseudo-labels and aligns cluster centroids across epochs to obtain more consistent supervision.
Source: Introduction · p. 2, Self-supervised Learning with Aligned… · p. 4Results
The method consistently achieves the best results among compared methods across all metrics and datasets.
Source: Results and Discussion · p. 6Takeaways & Limitations
With limited prior knowledge, the method obtains more accurate estimated cluster numbers and provides more stable, concrete supervision for clustering.
Source: Conclusion and Future Work · p. 7
Abstract
from arXiv · showhide
Discovering new intents is a crucial task in dialogue systems. Most existing methods are limited in transferring the prior knowledge from known intents to new intents. They also have difficulties in providing high-quality supervised signals to learn clustering-friendly features for grouping unlabeled intents. In this work, we propose an effective method, Deep Aligned Clustering, to discover new intents with the aid of the limited known intent data. Firstly, we leverage a few labeled known intent samples as prior knowledge to pre-train the model. Then, we perform k-means to produce cluster assignments as pseudo-labels. Moreover, we propose an alignment strategy to tackle the label inconsistency problem during clustering assignments. Finally, we learn the intent representations under the supervision of the aligned pseudo-labels. With an unknown number of new intents, we predict the number of intent categories by eliminating low-confidence intent-wise clusters. Extensive experiments on two benchmark datasets show that our method is more robust and achieves substantial improvements over the state-of-the-art methods. The codes are released at https://github.com/thuiar/DeepAligned-Clustering.
Introduction
The task is to discover new intents when unlabeled data mixes known and new intents, using limited labeled known-intent examples as prior knowledge. Deep Aligned Clustering combines that prior knowledge with aligned clustering signals and outperforms compared methods on two benchmark datasets.
Introduction
- Discovering new intents may reveal underlying user interests that provide business opportunities and guide service improvement.The paper frames intent discovery as important for improving dialogue-system service quality.Source: Introduction · p. 1
- Unlabeled data may mix known and new intents, so the task is to find known intents and discover new ones using limited labeled known-intent data.The prior CDAC+ approach uses ambiguous pairwise similarities, and performance drops as new intents increase.Source: Introduction · p. 1
- The approach pre-trains BERT intent representations with limited labeled data, then estimates an unknown cluster count by removing low-confidence clusters.It uses k-means assignments as pseudo-labels and aligns centroids across epochs to produce aligned labels for self-supervised learning.Source: Introduction · p. 2
- The method achieves better and more robust results than state-of-the-art methods on two benchmark datasets.It is designed to estimate novel-class counts and learn discriminative features with limited known-intent data.Source: Introduction · p. 2
Related Work
Related work spans intent modeling, unsupervised deep clustering, and semi-supervised clustering. The paper positions its alignment strategy against clustering methods that lack specific supervision or rely on pairwise signals.
Intent Modeling
- Intent-modeling work enriches intent information through joint tasks or uses hidden semantic information to construct supervised signals for feature learning.Source: Intent Modeling · p. 2
Unsupervised Clustering
- Deep clustering methods learn features alongside clustering, but many do not provide specific supervised signals; DeepCluster uses assignments as pseudo-labels but reinitializes its classifier each epoch.The proposed alignment strategy is presented as a way to produce aligned pseudo-labels without reinitialization.Source: Deep Clustering · p. 3
Semi-supervised Clustering
- Semi-supervised clustering transfers labeled-data knowledge, but CDAC+ lacks specific supervised signals and novel-class estimation, while DTC's performance depends strongly on labeled-data amount.Source: Semi-supervised Clustering · p. 3
Our Approach
The approach initializes BERT intent features from limited labeled known intents, then estimates the category count and iteratively clusters with aligned pseudo-label supervision. It matches centroids across epochs to stabilize assignments used to learn from unlabeled known and novel intents.
Intent Representation
- BERT’s last-layer token embeddings are mean-pooled and passed through dense Tanh layer h to form D-dimensional intent features.The pooled sentence representation is zi; the dense layer produces Ii.Source: Intent Representation · p. 3
Transferring Knowledge from Known Intents
- Cross-entropy pre-training on limited labeled intents initializes the feature extractor; the classifier is removed before unsupervised clustering.In each run, 75% of intents are randomly selected as known; for CLINC, the example split is 113 known and 37 new intents.Source: Pre-training · p. 3, Transferring Knowledge from Known… · p. 3, Table 1 · p. 4
- Starting from an oversized K′, the method estimates K by counting clusters at least threshold t, the expected cluster mean size N/K′.Clusters smaller than t are treated as low confidence and dropped.Source: Predict K · p. 4
Deep Aligned Clustering
- K-means assigns unlabeled intent features to centroids, and those assignments become pseudo-labels for feature learning.The assignments are produced from features extracted by the pre-trained model.Source: Unsupervised Learning by Clustering · p. 4
- Hungarian matching aligns current and prior-epoch centroids, mapping k-means assignments across epochs into pseudo-labels for softmax self-supervision.Centroids are used as the alignment targets because they summarize similar intent samples and are described as more stable guides.Source: Self-supervised Learning with Aligned… · p. 4
- The Silhouette Coefficient evaluates cluster quality during training by comparing within-cluster compactness with separation from the nearest other cluster.Higher SC scores indicate better clustering results.Source: Self-supervised Learning with Aligned… · pp. 4–5
Experiments
Experiments use CLINC and BANKING intent datasets, compare unsupervised and semi-supervised clustering methods, and evaluate clustering with NMI, ARI, and ACC under specified training settings.
Datasets
- CLINC contains 22,500 queries across 150 intents and 10 domains, while BANKING contains 13,083 customer-service queries with 77 intents.The experiments use these two benchmark intent datasets.Source: CLINC · p. 5, BANKING · p. 5, Datasets · p. 5
Baselines
- Unsupervised baselines include K-means, agglomerative clustering, SAE-KM, DEC, DCN, DAC, and DeepCluster; semi-supervised baselines include PCK-means, BERT-KCL, BERT-MCL, BERT-DTC, and CDAC+.The semi-supervised methods use the same BERT backbone as the proposed method for a fair comparison.Source: Unsupervised · p. 5, Semi-supervised · p. 6
- For unknown cluster counts, Table 4 evaluates predicted K at known-class ratios of 25%, 50%, and 75%, with K′ set to twice the ground-truth cluster count.Source: Table 4 · p. 5
Evaluation Metrics
- Clustering is evaluated with Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Accuracy (ACC), with ACC mappings obtained by the Hungarian algorithm.Source: Evaluation Metrics · p. 6
Evaluation Settings
- The setup labels 10% of training data and treats 75% of all intents as known; results are averaged over ten random-seed runs.The number of intent categories is set to ground truth, and Silhouette Coefficient is used during self-supervised learning evaluation.Source: Evaluation Settings · p. 6
Implementation Details
- The network uses 12-layer bert-uncased BERT with batch size 128, learning rate 5e−5, and 768-dimensional intent features; all but the last transformer layer are frozen.Source: Implementation Details · p. 6
Results and Discussion
Deep Aligned Clustering outperforms the compared baselines across metrics and datasets, while its alignment strategy and cluster-number estimation address specific challenges in intent discovery. Its results remain stronger as the known-class ratio changes and the assigned cluster count increases.
Results and Discussion
- The method consistently achieves the best results and outperforms other baselines by a large margin on all metrics and datasets.The authors also report that most semi-supervised methods outperform unsupervised methods.Source: Results and Discussion · p. 6
Effect of the Alignment Strategy
- The alignment strategy significantly improves results over reinitialization in both semi-supervised and unsupervised settings.The authors attribute this to mapping pseudo-labels across consecutive epochs, which retains embedding history and provides stronger supervised signals.Source: Effect of the Alignment Strategy · p. 6
Estimate K
- With known-class ratios of 25%, 50%, and 75%, the method achieves almost the lowest cluster-number error rates and estimates K by removing low-confidence clusters.BERT-DTC is a strong but unstable baseline, while BERT-MCL's predicted K tends toward the number of known classes.Source: Estimate K · pp. 6–7
Effect of the Known Class Ratio
- Across known-class ratios of 25%, 50%, and 75%, the method achieves the best results and greater robustness with fewer known intents.BERT-MCL and BERT-DTC drop sharply as the known-class ratio decreases; the authors say their learned features become more biased toward labeled data, whereas their method uses labels only to learn feature representations and avoids that bias during self-supervised learning.Source: Effect of the Known Class Ratio · p. 7
Effect of the Number of Clusters
- When assigned K′ ranges from the ground-truth class count to four times that count, the method achieves the best results across cluster counts.The authors report that many semi-supervised methods are vulnerable to larger K′, which can split fine-grained clusters sharing an intent label; they attribute their method's performance to more accurate cluster-number estimation.Source: Effect of the Number of Clusters · p. 7, Figure 5 · p. 7, Figure 6 · p. 7
Conclusion and Future Work
The conclusion presents Deep Aligned Clustering as a method for using limited known-intent data to discover new intents and estimate their number. Experiments on two benchmarks show improvements over compared methods, and future work will explore other clustering and self-supervised approaches.
Conclusion and Future Work
- The method transfers prior knowledge from limited known intents and estimates intent count by eliminating low-confidence clusters.It also provides more stable and concrete supervised signals to guide clustering.Source: Conclusion and Future Work · p. 7
- On two challenging benchmark datasets, the method improves over compared methods and estimates cluster numbers more accurately with limited prior knowledge.Source: Conclusion and Future Work · p. 7
- Future work will explore different clustering methods for producing supervised signals and other self-supervised methods for representation learning.Source: Conclusion and Future Work · p. 7