PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinement

Ting-En Lin, Hua Xu, Hanlei Zhang

arXiv:1911.08891v1cs.CLcs.LG

TL;DR

New-intent discovery must cluster vast, noisy unlabeled conversational data despite limited labels and unknown intent categories. CDAC+ uses an auxiliary target distribution P to learn from high confidence assignments, refining model parameters and cluster centroids. CDAC+ yields significant improvements over strong baselines on three datasets; on StackOverflow, it increases clustering accuracy over DAC+ by up to 50%.

  • Problem

    New-intent discovery must cluster vast, noisy unlabeled conversational data despite limited labels and unknown intent categories.

    Source: Introduction · p. 1

  • Method

    CDAC+ uses an auxiliary target distribution P to learn from high confidence assignments, refining model parameters and cluster centroids.

    Source: Cluster Refinement with KLD loss · p. 4

  • Results

    CDAC+ yields significant improvements over strong baselines on three datasets; on StackOverflow, it increases clustering accuracy over DAC+ by up to 50%.

    Source: Introduction · p. 2, Ablation study · p. 6

  • Takeaways & Limitations

    CDAC+ is insensitive to cluster count, maintaining strong performance as predefined clusters increase from the ground-truth count to four times that count.

    Source: Effect of Number of Clusters · p. 6

    Some SearchCreativeWork samples are incorrectly assigned to SearchScrrenEvent clusters because the intents are semantically similar.

    Source: Error Analysis · p. 8

Abstract

from arXiv · show

Identifying new user intents is an essential task in the dialogue system. However, it is hard to get satisfying clustering results since the definition of intents is strongly guided by prior knowledge. Existing methods incorporate prior knowledge by intensive feature engineering, which not only leads to overfitting but also makes it sensitive to the number of clusters. In this paper, we propose constrained deep adaptive clustering with cluster refinement (CDAC+), an end-to-end clustering method that can naturally incorporate pairwise constraints as prior knowledge to guide the clustering process. Moreover, we refine the clusters by forcing the model to learn from the high confidence assignments. After eliminating low confidence assignments, our approach is surprisingly insensitive to the number of clusters. Experimental results on the three benchmark datasets show that our method can yield significant improvements over strong baselines.

Introduction

New-intent discovery must work with sparse labels and unknown categories, while intent taxonomies and cluster counts are difficult to specify. CDAC+ uses labeled pairwise constraints and confidence-based refinement, with reported gains over strong baselines.

Introduction

  • Intent discovery is difficult because the number of new intents is hard to estimate and heuristic taxonomies may group utterances by the wrong criteria.For example, data intended to cluster by technical problems may instead cluster by question type.Source: Introduction · p. 1
  • Existing approaches require intensive feature engineering and often separate representation learning from cluster assignment in a pipeline.Source: Introduction · p. 1
  • The setting combines limited labeled data with vast unlabeled data, unknown intent categories, and unlabeled examples containing both known and unknown intents.Source: Introduction · p. 1
  • CDAC+ uses BERT representations and labeled pairwise constraints to guide end-to-end clustering, then refines clusters using high-confidence assignments.The method frames clustering as pairwise classification and eliminates low-confidence assignments.Source: Introduction · pp. 1–2
  • Experiments on three datasets show that CDAC+ yields significant improvements over strong baselines.Source: Introduction · p. 2

Related Work

Prior work approaches clustering through transfer learning, deep representation learning, or pairwise constraints. CDAC+ transfers pretrained representations and data relations while using pairwise constraints in its clustering objective.

Transfer Learning

  • CDAC+ transfers data relations through pairwise similarity and pretrained-model parameters without an extra similarity-prediction model.Unlike few-shot learning, its clustering setting includes both known and unknown classes at test time and transfers knowledge through a classification approach [Snell, Swersky, and Zemel 2017].Source: Transfer Learning · p. 2, Few-shot learning · p. 2

Unsupervised Clustering

  • Traditional clustering struggles with high-dimensional data, while DEC and DCN learn representations and assignments but do not effectively capture context.DAC uses pairwise classification for adaptive clustering, but its CNN-based similarity assumption does not work for text; CDAC+ replaces CNN with a pretrained language model.Source: Unsupervised Clustering · p. 2, Deep Neural Network-based Clustering · p. 3

Constrained Clustering

  • Constrained-clustering methods use hard or soft pairwise constraints, and CDAC+ uses pairwise constraints to optimize the clustering objective and metric learning.Source: Constrained Clustering · p. 3

Constrained Deep Adaptive Clustering

CDAC+ combines learned intent representations, pairwise constraints, and cluster refinement. Its refinement stage uses high-confidence assignments to update model parameters and cluster centroids.

with Cluster Refinement

Intent Representation

Pairwise-Classification with Similarity Loss

  • Labeled sentence pairs provide pairwise constraints that guide how the model partitions the data.Source: Supervised Step · p. 3

with Cluster Refinement

  • Each experimental run treats 25% of intents as unknown; for SNIPS, that means two unknown and five known intents.Source: Table 1 · p. 4

Pairwise-Classification with Similarity Loss

  • Dynamic similarity thresholds label selected pairs as similar or dissimilar, while pairs between thresholds are excluded from training.Source: Self-supervised Step · pp. 3–4
  • The adaptive threshold parameter is increased during training to select more sentence pairs, potentially adding noise to the self-labeled matrix.Source: Self-supervised Step · p. 4

Cluster Refinement with KLD loss

  • An expectation-maximization refinement iteratively uses soft assignments and a target distribution to focus learning on high-confidence assignments.The iterations stop when cluster assignments change by less than δlabel% in two consecutive iterations.Source: Cluster Refinement with KLD loss · p. 4

Experiments

Experiments on three short-text datasets compare CDAC+ with unsupervised and semi-supervised clustering methods, and report significant overall gains. Further analyses examine cluster-count robustness, the effects of constraints and refinement, and performance under different data conditions.

Datasets

  • Experiments use SNIPS, DBPedia, and StackOverflow; the latter two are sampled at 1,000 examples per class.SNIPS has 14,484 utterances across seven intent types; the cited dataset descriptions give 14 DBPedia classes and 3,370,528 StackOverflow titles across 20 classes.Source: Datasets · p. 4, SNIPS · p. 4, DBPedia (Zhang and LeCun 2015) · p. 4

Results and Discussion

  • CDAC+ outperforms other baselines by a significant margin across all datasets and evaluation metrics.The reported results attribute its grouping performance to intent representations learned with pairwise classification and constraints, and report generalization to intents unknown in advance.Source: Table 2 · p. 5, Results and Discussion · pp. 5–6

Baselines

Evaluation Metrics / Experimental Settings

  • Clustering is evaluated with NMI, ARI, and ACC over ten runs, using 25% unknown classes and 10% labeled training data in each run.The evaluation tunes on known-intent validation data and reports the Hungarian-aligned clustering accuracy.Source: Evaluation Metrics · p. 5, Experimental Settings · p. 5

Results and Discussion

  • Adding constraints improves most methods, while CDAC+ and DAC+ outperform representation-only variants, supporting the contribution of cluster refinement.On StackOverflow, CDAC+ increases clustering accuracy by up to 50% over DAC+; the authors state that refinement is needed for better results.Source: Table 3 · p. 5, Ablation study · p. 6
  • CDAC+ remains comparatively robust as the predefined cluster count rises from the ground-truth count to four times that count.The authors report that almost all other methods drop dramatically and that CDAC+ is insensitive to cluster count.Source: Effect of Number of Clusters · p. 6, Figure 3 · p. 6, Figure 7 · p. 7
  • CDAC+ performs better than most baselines even with labeled-data ratios below 0.1, while remaining robust as unknown-class ratios increase.On StackOverflow, the labeled-data effect is largest; BERT-Semi's NMI is slightly higher at ratios 0.01 and 0.03, while its performance drops dramatically as unknown classes increase.Source: Figure 4 · p. 6, Figure 5 · p. 6, Effect of Labeled Data · p. 7
  • CDAC+ is robust to class imbalance, whereas other baselines lose around 3% to 10% under different minimum-retention probabilities.The study also reports compact within-class and separable between-class representations, and finds that BookRestaurant and SearchCreativeWork can be discovered when unknown during training, though some semantically similar examples are confused.Source: Performance on Imbalanced Dataset · p. 7, Figure 8 · p. 7, Error Analysis · pp. 7–8

Conclusion and Future Work

CDAC+ uses limited labeled data to guide new-intent clustering and refines clusters using high-confidence assignments; experiments report improvements over strong baselines and insensitivity to cluster count.

Loading 1911.08891v1…