PaperlayerView exact revision on arXiv

Source-linked AI summary

A Co-Interactive Transformer for Joint Slot Filling and Intent Detection

Libo Qin, Tailu Liu, Wanxiang Che, Bingbing Kang, Sendong Zhao, Ting Liu

arXiv:2010.03880v3cs.CL

TL;DR

Intent detection and slot filling are closely related SLU tasks, yet prior work did not simultaneously model their bidirectional interaction. The paper introduces a Co-Interactive Transformer with cross-task attention, achieving state-of-the-art results on SNIPS and ATIS.

  • Problem

    Prior SLU approaches either model intent detection and slot filling separately or use only one-way information flow, despite the tasks being closely related.

  • Method

    The Co-Interactive Transformer uses label attention and bidirectional co-interactive attention to exchange information between intent and slot representations.

  • Results

    The framework achieves state-of-the-art performance on the SNIPS and ATIS benchmarks, with significant improvement over baselines.

  • Takeaways & Limitations

    Experiments support that modeling mutual interaction enables the framework to capture useful shared knowledge between intent detection and slot filling.

Abstract

from arXiv · show

Intent detection and slot filling are two main tasks for building a spoken language understanding (SLU) system. The two tasks are closely related and the information of one task can be utilized in the other task. Previous studies either model the two tasks separately or only consider the single information flow from intent to slot. None of the prior approaches model the bidirectional connection between the two tasks simultaneously. In this paper, we propose a Co-Interactive Transformer to consider the cross-impact between the two tasks. Instead of adopting the self-attention mechanism in vanilla Transformer, we propose a co-interactive module to consider the cross-impact by building a bidirectional connection between the two related tasks. In addition, the proposed co-interactive module can be stacked to incrementally enhance each other with mutual features. The experimental results on two public datasets (SNIPS and ATIS) show that our model achieves the state-of-the-art performance with considerable improvements (+3.4% and +0.9% on overall acc). Extensive experiments empirically verify that our model successfully captures the mutual interaction knowledge.

1. INTRODUCTION

SLU jointly addresses intent detection and slot filling, but prior approaches either model their relationship implicitly or use only one-way intent-to-slot information. The Co-Interactive Transformer instead models bidirectional cross-impact between the tasks and improves performance on SNIPS and ATIS.

  • 1. INTRODUCTION: Prior joint models either share an encoder to model task relationships implicitly or guide slot filling with intent information only.The latter approaches do not model information flow from slots back to intent detection.
  • 1. INTRODUCTION: The Co-Interactive Transformer models bidirectional cross-impact between intent detection and slot filling through a dedicated co-interactive module.Intent and slot representations interact in both directions to mutually enhance the two tasks.
  • 1. INTRODUCTION: The framework achieves significant improvement over all baselines on the SNIPS and ATIS benchmark datasets.The BERT-enhanced framework reaches new state-of-the-art performance according to the paper.

2. APPROACH

The approach combines a shared BiLSTM encoder with a co-interactive module that explicitly connects intent and slot representations. Label attention extracts task-specific semantics, and bidirectional attention then updates each representation using the other.

  • 2. APPROACH: The framework consists of a shared encoder, a co-interactive module, and separate decoders for intent detection and slot filling.The shared encoder produces contextual representations that feed the interaction module and task-specific decoders.
  • 2.1. Shared Encoder: BiLSTM encodes each input sequence in both directions to produce context-sensitive hidden states.The encoder uses forward and backward LSTM layers to capture temporal features within word orders.
  • 2.2. Co-Interactive Module: The co-interactive module first applies intent and slot label attention, then replaces self-attention with explicit mutual interaction.A subsequent feed-forward stage further fuses intent and slot information implicitly.
  • 2.2. Co-Interactive Module: Label attention uses decoder parameters as intent and slot label embedding matrices to obtain explicit task representations.These representations capture intent and slot semantic information before cross-task interaction.
  • 2.2. Co-Interactive Module: Cross-attention updates slot representations with intent information and intent representations with slot information, establishing a bidirectional connection.The module uses task-specific queries, keys, and values to align each task with the other.

Intent-Aware Slot and Slot-Aware Intent Representation

The framework explicitly aligns slot and intent representations through cross-task attention, then fuses them before decoding. Stacking these modules repeatedly updates both representations with information from the other task.

  • Intent-Aware Slot and Slot-Aware Intent Representation: Cross-attention uses slot queries with intent keys and values to incorporate corresponding intent information into slot representations.The reverse direction uses intent queries with slot keys and values to obtain slot-aware intent representations.
  • Intent-Aware Slot and Slot-Aware Intent Representation: The model concatenates updated intent and slot representations before feed-forward fusion, producing aligned features for both tasks.The resulting updated representations align corresponding slot and intent features.
  • Intent-Aware Slot and Slot-Aware Intent Representation: Stacking multiple co-interactive layers provides repeated interaction between the two tasks before the final intent and slot representations are obtained.The framework applies a stacked co-interactive attention network with multiple layers.
  • Intent-Aware Slot and Slot-Aware Intent Representation: The updated intent representation supports max-pooling for intent detection, while a CRF models dependencies among slot labels.Intent prediction uses a sentence representation; slot filling uses a standard CRF layer.

3. EXPERIMENTS

Experiments evaluate joint intent detection and slot filling on ATIS and SNIPS using standard task metrics and compare the proposed interaction mechanisms with ablations and baselines. The results support explicit bidirectional interaction over one-way or implicit interaction designs.

  • 3. EXPERIMENTS: The experiments use ATIS and SNIPS, jointly optimize intent detection and slot filling, and evaluate F1, accuracy, and overall accuracy.Table 1 reports slot filling and intent detection results on the two datasets.
  • 3. EXPERIMENTS: 6.2% and 2.1% overall acc improvements over CM-Net are reported on SNIPS and ATIS, respectively.The comparison concerns the proposed framework and CM-Net.
  • 3. EXPERIMENTS: Removing either intent or slot attention reduces slot filling and intent detection performance, indicating that both explicit representations contribute to the co-interactive layer.These are ablations reported in Table 2.
  • 3. EXPERIMENTS: The co-interactive framework outperforms self-attention, which models task interaction implicitly rather than explicitly.The self-attention variant concatenates the intent and slot representations before applying self-attention.

Bidirectional Connection vs. One Direction Connection

The framework's bidirectional interaction outperforms variants that retain only intent-to-slot or slot-to-intent information flow. The authors attribute this improvement to mutual interaction between slot filling and intent detection.

  • The framework outperforms both intent-to-slot and slot-to-intent variants.These variants retain only one direction of information flow by using one representation type to query the other.
  • The authors attribute the improvement to modeling mutual interaction between slot filling and intent detection.They contrast this with single-direction models, which cannot capture the two-way interaction.

4. RELATED WORK

Prior SLU research used separate task models, implicit multi-task sharing, or explicit one-way intent guidance. The proposed co-interactive Transformer instead establishes directional connections that model cross-impact between the two tasks.

  • Earlier methods addressed intent detection and slot filling with classifiers such as SVM or RNN, and slot filling with CRF or RNN models.
  • Joint models commonly represented task relations implicitly through shared parameters or explicitly by using intent information to guide slot filling.
  • The proposed co-interactive Transformer simultaneously models cross-impact through directional connections between the two tasks.
  • Compared with iterative mutual-interaction models, the framework builds a bidirectional connection simultaneously within one unified framework.

5. CONCLUSION

The paper proposes a co-interactive Transformer for joint slot filling and intent detection that exploits mutual interaction knowledge. Experiments on two datasets support the framework's effectiveness and state-of-the-art performance.

  • The proposed co-interactive Transformer uses mutual interaction knowledge for joint slot filling and intent detection.
  • Experiments on two datasets show the effectiveness of the proposed models.
  • The framework achieves state-of-the-art performance.
Loading 2010.03880v3…