PaperlayerView exact revision on arXiv

Source-linked AI summary

Global-to-local Memory Pointer Networks for Task-Oriented Dialogue

Chien-Sheng Wu, Richard Socher, Caiming Xiong

arXiv:1901.04713v2cs.CLcs.AI

TL;DR

End-to-end task-oriented dialogue struggles to incorporate large, dynamic knowledge bases reliably. GLMP shares external knowledge between a global memory encoder and local memory decoder, and it improves results over prior state-of-the-art models on simulated bAbI and human-human Stanford Multi-domain Dialogue evaluations.

  • Problem

    Large, dynamic knowledge bases are difficult to incorporate into end-to-end task-oriented dialogue because they are noisy inputs that are hard to encode and decode reliably.

  • Method

    GLMP uses a global memory encoder and local memory decoder that share external knowledge, filter it with a global pointer, and instantiate sketch-response slots with local pointers.

  • Results

    GLMP improves over previous state-of-the-art models on both simulated bAbI Dialogue and human-human Stanford Multi-domain Dialogue evaluations.

  • Takeaways & Limitations

    GLMP can produce task-oriented responses in out-of-vocabulary settings while copying relevant entities from external knowledge.

Abstract

from arXiv · show

End-to-end task-oriented dialogue is challenging since knowledge bases are usually large, dynamic and hard to incorporate into a learning framework. We propose the global-to-local memory pointer (GLMP) networks to address this issue. In our model, a global memory encoder and a local memory decoder are proposed to share external knowledge. The encoder encodes dialogue history, modifies global contextual representation, and generates a global memory pointer. The decoder first generates a sketch response with unfilled slots. Next, it passes the global memory pointer to filter the external knowledge for relevant information, then instantiates the slots via the local memory pointers. We empirically show that our model can improve copy accuracy and mitigate the common out-of-vocabulary problem. As a result, GLMP is able to improve over the previous state-of-the-art models in both simulated bAbI Dialogue dataset and human-human Stanford Multi-domain Dialogue dataset on automatic and human evaluation.

1 INTRODUCTION

GLMP addresses the difficulty of incorporating large, dynamic knowledge bases into end-to-end task-oriented dialogue. It combines global and local memory pointers to filter external knowledge, generate sketch responses, and copy slot values, improving performance in OOV and human-human dialogue settings.

  • Motivation: Large, dynamic knowledge bases are difficult to encode and decode, making response generation unstable and risking incorrect entities in task-oriented dialogue.The problem is especially consequential when responses require precise knowledge-base entities, such as addresses.
  • Approach: GLMP combines a global memory encoder, a local memory decoder, and shared external knowledge.The architecture is designed to incorporate external knowledge into an end-to-end learning framework.
  • Approach: The global memory pointer softly filters unnecessary external-knowledge words before the local decoder copies values into sketch response tags.The decoder first produces a sketch response without slot values, then instantiates the tags from relevant knowledge-base content.
  • Results: 92.0% per-response accuracy and a 7.5% full-dialogue improvement over existing end-to-end approaches are reported on simulated OOV bAbI Dialogue tasks.These results are reported for the simulated out-of-vocabulary setting.
  • Results: GLMP surpasses the previous state of the art on both automatic and human evaluation in the human-human Stanford Multi-domain Dialogue dataset.The reported evaluation supports the effectiveness of combining global and local memory pointers across both simulated and human-human settings.

2 GLMP MODEL

GLMP combines a global memory encoder, shared external knowledge, and a local memory decoder to generate task-oriented responses through global filtering and local copying. The model encodes dialogue context, produces a sketch response with slot tags, and fills those slots using local memory pointers.

  • 2 GLMP MODEL: GLMP consists of a global memory encoder, shared external knowledge, and local memory decoder that jointly transform dialogue history and KB information into responses.The dialogue history and KB are inputs, while the system response is the expected output.
  • 2.1 EXTERNAL KNOWLEDGE: The external knowledge stores word-level information from both structured KB entries and temporal dialogue history in separate memory modules.KB elements use subject-relation-object triplets, while dialogue context includes speaker and temporal encodings.
  • 2.2 GLOBAL MEMORY ENCODER: The context RNN writes dialogue-history hidden states into external knowledge, then uses the final hidden state to read the memory and produce a global memory pointer.The global pointer is passed to the decoder, while the final memory readout provides encoded KB information.
  • 2.2 GLOBAL MEMORY ENCODER: The global memory pointer uses independent sigmoid probabilities to softly filter memory positions rather than normalized attention weights.An auxiliary multi-label classification loss provides additional supervision for this pointer.
  • 2.3 LOCAL MEMORY DECODER: The local decoder first generates a sketch response without slot values, using sketch tags such as @poi or @distance.The sketch RNN is trained from encoded dialogue and KB information and predicts both vocabulary tokens and queries to external knowledge.
  • 2.3 LOCAL MEMORY DECODER: At each decoding step, the global pointer filters contextual memory and the local memory pointer selects copied text to replace generated sketch tags.The local pointer is the final-hop memory distribution and is supervised with position labels, including a null token when the target is absent from memory.

3 EXPERIMENTS

Experiments evaluate GLMP on bAbI Dialogue and SMD, showing strong OOV robustness, task completion, entity prediction, and human-rated response quality. Ablations and visualization examine how the global pointer and dialogue-history writing contribute to these results.

  • Datasets: GLMP is evaluated on simulated bAbI Dialogue and human-human Stanford Multi-domain Dialogue datasets.bAbI includes OOV entity-value test sets, while SMD covers calendar, weather, and navigation domains with more complex knowledge bases.
  • bAbI Dialogue: GLMP achieves 92.0% task-completion rate in the full bAbI dialogue task and has the least OOV performance drop.Its task-5 performance decreases by 7–9% in the OOV setting, while earlier generation-based approaches show larger drops in simpler tasks.
  • Stanford Multi-domain Dialogue: 14.79 BLEU and 59.97% entity F1 are the highest reported GLMP scores on SMD among the compared models.The entity F1 improvement is described as substantial, while the BLEU improvement is slight; some rule-based and KVR results use simplified canonicalized entities.
  • Human Evaluation: Human evaluation on 200 test scenarios finds GLMP more appropriate and human-like than Mem2Seq on both measures.Amazon Mechanical Turk raters scored appropriateness and human-likeness from 1 to 5, with original dataset responses also included for comparison.
  • Ablation Study: Removing dialogue-history writing causes 5.3% more loss in full-dialogue bAbI performance, while removing global weighting drops SMD entity F1 by 11.47%.A 0.4% increase in bAbI task 5 without the global pointer suggests that its entity prior can sometimes be too strong.
  • Visualization and Qualitative Evaluation: Attention visualization shows the final pointer copying the entity chevron and address 783 Arcadia Pl into sketch-response slots.The global pointer mitigates attention to several competing points of interest and addresses.

4 RELATED WORKS

Related work places GLMP among modular and end-to-end task-oriented dialogue systems, pointer-based copying methods, and memory-augmented neural networks. Its distinguishing design uses shared external knowledge with global-to-local pointer processing and multiple-hop attention.

  • Task-oriented dialogue systems: Task-oriented dialogue research includes modular pipelines and end-to-end systems, with the latter generating responses token by token.The related work covers modules for understanding, state tracking, management, and generation, alongside end-to-end approaches.
  • Memory pointer visualization: Figure 3 compares global memory attention, attention without global weighting, and the final memory pointer in SMD navigation.The visualization makes the global-to-local memory-processing design directly comparable across three attention views.
  • Pointer network: Pointer networks use attention to select input members as outputs, and copy mechanisms have been applied across several NLP tasks.Applications include question answering, neural machine translation, language modeling, text summarization, and task-oriented dialogue.
  • Others: Other approaches simplify entity handling through entity indexing or manually recorded delexicalization of entity tables.These methods are discussed alongside memory controllers that query external memory recurrently.
  • Memory-augmented models: GLMP differs from single-matrix memory models by using end-to-end memory networks for multiple-hop attention.The approach is related to memory-augmented neural networks and stacked self-attention in the Transformer.

5 CONCLUSION

The paper presents GLMP as an end-to-end task-oriented dialogue model that shares external knowledge between global encoding and local decoding. It reports effective responses in OOV settings and state-of-the-art results on simulated and human-human datasets.

  • Conclusion: GLMP combines a global memory encoder, shared external knowledge, and a local memory decoder for end-to-end task-oriented dialogue.The paper also visualizes how the global pointer helps and identifies possible extensions to question answering and text summarization.
  • Conclusion: GLMP produces effective responses in OOV scenarios and achieves state-of-the-art results on both simulated and human-human dialogue datasets.The conclusion attributes this outcome to the global and local memory pointers.

A.1 TRAINING PARAMETERS

Table 5 lists selected hyper-parameters for each dataset and memory-hop setting, including embedding dimensions, GRU hidden sizes, dropout rates, learning rate, and decay rate.

  • Table 5 reports embedding dimensions, GRU hidden sizes, dropout rates, and memory-hop settings for each dataset.
  • The models use a learning rate of 0.001 with a decay rate of 0.5.

A.2 DATASET STATISTICS

Table 6 presents dataset statistics for the two datasets.

  • Table 6 summarizes dataset statistics for two datasets.

A.3 HUMAN EVALUATION

Human evaluation measures dialogue appropriateness and human-likeness across 200 dialogue scenarios. Appropriateness assesses correctness and dialogue quality, while human-likeness assesses perceived naturalness.

  • Human evaluation reports appropriateness and human-likeness scores across 200 dialogue scenarios.
  • Appropriateness: Appropriateness measures grammar, logic, dialogue flow, and correctness of the provided entity.
  • Appropriateness: The appropriateness scale ranges from 1 for wholly incorrect responses to 5 for responses with correct grammar, logic, flow, and entities.
  • Human-Likeness: Human-likeness rates how closely an utterance resembles what a person would say on a 1-to-5 scale.

B ERROR ANALYSIS

Error analysis identifies task- and dataset-specific failure modes, while noting that rule-based, human-labeled pipeline systems may be more robust under the available data conditions.

  • In bAbI, errors mainly occur in task 3 when recommending restaurants by rating after users reject higher-rated options.
  • SMD errors arise when multiple knowledge-base options match an intention, especially when several doctor appointments are present.
  • Without domain-specific or user-intention supervision, incorrect delexicalized responses can produce wrong entity copies or mismatched sketch tags.
  • Given the datasets' limited size, carefully constructed PyDial systems with human rules may outperform end-to-end systems in known domains.

D VISUALIZATION

The visualizations pair delexicalized response sketches with final slot-filled generations across SMD navigation, schedule, and weather domains. Examples show both successful knowledge grounding and a failure to resolve multiple appointment options.

  • Navigation: Delexicalized sketches are instantiated with concrete entities, distances, and addresses in navigation examples.The generated responses fill placeholders such as @poi, @distance, and @address with database values.
  • Navigation: Several final generations closely match the available gold responses for nearby places and addresses.Examples include Willows Market, Stanford Express Care, and Civic Center Garage, though one address differs from the gold response.
  • Schedule: A schedule-domain example fails to resolve which doctor appointment the user means when multiple appointments exist.The generation selects one appointment although the gold response asks whether the user means the Wednesday appointment or the one on the 5th.
  • Schedule: The appointment ambiguity matches a broader SMD difficulty when several knowledge-base options correspond to one user intention.The reported issue is that the model can barely recognize which appointment is intended when multiple doctor appointments appear in the table.
Loading 1901.04713v2…