PaperlayerView exact revision on arXiv

Source-linked AI summary

Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems

Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, Kaheer Suleman

arXiv:1704.00057v2cs.CL

TL;DR

Goal-oriented dialogue systems often overwrite constraints, limiting comparison and backtracking. The paper introduces Frames, a human-human dialogue corpus, and formalizes frame tracking to study simultaneous memory across contexts. It reports that simple baseline rules are far from adequate and identifies further uses for studying dialogue management and information presentation.

  • Problem

    Sequential dialogue systems overwrite earlier constraints, limiting users’ ability to compare results or backtrack during decision-making.

  • Method

    The paper collects human-human Wizard-of-Oz dialogues and formalizes frame tracking, which simultaneously maintains multiple semantic frames.

  • Results

    The rule-based baseline is only slightly better than random for frame identification and similar to random for frame creation, indicating substantial room for improvement.

  • Takeaways & Limitations

    Frames supports research on memory in dialogue management and information presentation through natural language generation.

Abstract

from arXiv · show

This paper presents the Frames dataset (Frames is available at http://datasets.maluuba.com/Frames), a corpus of 1369 human-human dialogues with an average of 15 turns per dialogue. We developed this dataset to study the role of memory in goal-oriented dialogue systems. Based on Frames, we introduce a task called frame tracking, which extends state tracking to a setting where several states are tracked simultaneously. We propose a baseline model for this task. We show that Frames can also be used to study memory in dialogue management and information presentation through natural language generation.

1 INTRODUCTION

Frames targets a limitation of sequential goal-oriented dialogue systems: they overwrite earlier constraints, preventing comparison and backtracking. It introduces a memory-focused corpus and frame tracking, which maintains multiple semantic frames simultaneously.

  • Traditional systems overwrite earlier constraints, so users cannot compare results or move back and forth between alternatives.
  • E-commerce users may compare and explore options, requiring systems to separately remember properties associated with different items or preferences.
  • Frames contains 1369 human-human Wizard-of-Oz dialogues about finding vacation packages using constraints such as destination and budget.
  • Frame tracking requires agents to track multiple semantic frames throughout a dialogue rather than compressing its history into one frame.
  • The paper formalizes frame tracking, provides a baseline model, and positions Frames for studying memory in dialogue management and information presentation.

2 MOTIVATION

Conversational booking involves comparison and backtracking that simple slot-filling systems and conventional interfaces handle poorly. Frames is proposed to support text-based agents that remember options while users make decisions.

  • Simple chatbot tasks use sequential slot-filling, whereas flight booking involves comparison and backtracking during search.
  • Graphical interfaces make comparison difficult because limited visual space and hierarchical vacation packages complicate displaying multiple options and properties.
  • A text interface could complement graphical interfaces by remembering user-mentioned properties and displaying comparisons on request.
  • The Frames dataset supports text-based conversational agents that help users make decisions by remembering every option explored during the dialogue.

3 DATA COLLECTION

Frames was collected through human-human text dialogues in a Wizard-of-Oz setting, using templated vacation-search tasks and controlled but varied wizard behavior. The design supports realistic memory, information-presentation, and dialogue-management behaviors.

  • The corpus was collected over 20 days with 12 participants paired through a chat interface.
  • 3.1 WIZARD-OF-OZ DATA COLLECTION: Wizard-of-Oz collection lets a human act as the dialogue system with database access, avoiding the need for preexisting dialogue-system components.
  • 3.1 WIZARD-OF-OZ DATA COLLECTION: Text-based human-human dialogues enabled realistic behaviors unavailable to current architectures while avoiding speech-recognition noise and time-sensitive turn-taking.
  • 3.2 TASK TEMPLATES AND INSTRUCTIONS: Tasks were generated from templates specifying vacation constraints, with controlled probabilities of successful and unsuccessful database queries.
  • 3.2 TASK TEMPLATES AND INSTRUCTIONS: The collection used 38 templates, including generic and role-playing variants, to increase behavioral variety and reduce repetition.
  • 3.2 TASK TEMPLATES AND INSTRUCTIONS: Wizard instructions permitted varied answers and occasional bad behavior, enabling analysis of cooperativity and user satisfaction.
  • 3.2 TASK TEMPLATES AND INSTRUCTIONS: Wizards could choose whether to present suggestions formed by relaxing constraints after a query returned no results, supporting study of recommendation timing.

4 STATISTICS OF THE CORPUS

Frames comprises 1369 dialogues and 19986 turns averaging 15 turns each, with multi-act turns, user ratings, and annotated dialogue-act distributions characterizing the corpus.

  • 1369 dialogues contain 19986 turns, averaging 15 turns per dialogue.
  • About 25% of dialogue turns contain more than one dialogue act.
  • Turns with zero dialogue acts correspond to user requests the wizard could not provide and were usually followed by an inability response.
  • More than 70% of dialogues received the maximum user rating of 5.

5 DIALOGUE ANNOTATION SCHEME

The annotation scheme records dialogue acts, slot information, active frames, and relationships among frames to preserve evolving user goals and comparisons. Frames are created, switched, inherited, and cross-referenced throughout each dialogue.

  • Annotation labels: The dataset records dialogue acts, slot types, slot values, references to other frames, the active frame, and automatically computed frame labels.These annotations combine utterance-level information with frame-level state.
  • Semantic frames: Semantic frames contain user requests, binary questions, constraints, and comparison requests.Constraints may be set by either the user or the wizard.
  • Frame creation and switching: New frames begin at frame 1 and are introduced when the wizard offers or suggests an item or the user modifies established slots.The active frame changes only when the user switches back to a frame, allowing earlier constraint sets to remain available.
  • Cross-references: Frame ids identify frames, while ref, read, and write annotate cross-references and the movement of slot values between frames.Read indicates the source frame; write indicates where values are stored.
  • Annotation example: In an example with two flight offers, later frames inherit values from the first hotel offer while adding different seat classes and prices.The offers distinguish business-class travel at 1002.27 USD from economy travel at 812.69 USD.
  • Corpus statistics: 6.71 frames and 3.58 frame switches occur on average per dialogue.Figure 2 presents boxplots for frame creations and frame changes across the corpus.

6 RESEARCH TOPICS

Frames supports research on memory beyond single-state tracking by representing multiple goals, frame references, comparisons, and database-aware information presentation. The paper proposes frame tracking and identifies dialogue-management and generation challenges exposed by the corpus.

  • Frame tracking: Frame tracking extends state tracking by maintaining several semantic frames simultaneously instead of overwriting earlier slot values.This preserves options for comparison and return to previously discussed items.
  • Frame tracking: The task requires detecting frame creation, identifying the active frame, and predicting referenced frames and their dialogue-act associations.Frame identification and frame creation are evaluated as separate metrics.
  • Frame tracking: 979 switch-frame acts and 455 request-compare acts require resolving references from text and anaphora, even when users omit explicit slot values.The example “Which package has the soonest departure?” refers to multiple packages indirectly.
  • Related work: Unlike related task-lineage work, Frames contains naturally occurring topic switches and comparisons rather than artificially mixed tasks alone.The cited framework allows task switching but does not handle comparisons.
  • Dataset contribution: The dataset is intended to encourage research on complex state-tracking behaviours and provides frame tracking as a principled decision-making formulation.Researchers may also define other tasks over the corpus.
  • Dialogue management: Dialogue management must sometimes retrieve and act on non-active frames, while multi-action turns challenge reinforcement-learning-based management.Wizards also summarize database results, requiring reasoning about how to present information.

7 DATASET FORMAT

The dataset is distributed as JSON with dialogue-level metadata, turn-level annotations, frame histories, and package-related hotel information. Its format records both the active conversational context and accumulated frame contents.

  • 7.1 DIALOGUES: Each dialogue contains turns, labels, user and wizard identifiers, and a unique dialogue id.
  • 7.1 DIALOGUES: The labels record userSurveyRating on a 1–5 scale and the wizard’s perceived task completion.
  • 7.1 DIALOGUES: Each turn stores its author, utterance text, active frame id, dialogue acts, timestamps, and all frames after the current turn is integrated.
  • 7.1 DIALOGUES: Frame records include parent-frame links, user requests and questions, comparison requests, and frame information properties.
  • 7.1 DIALOGUES: Slot values accumulate within a frame, may have multiple values, and can be marked as negated.
  • 7.2 HOTELS: Vacation packages are randomly generated, and hotel descriptions list only true amenities or nearby points of interest.
  • 7.2 HOTELS: Figure 3 summarizes hotel amenities per hotel and the number of nearby hotels for points of interest.

8 BASELINES

The paper evaluates baselines for natural language understanding and frame tracking. The NLU model jointly predicts acts and slots, while rule-based frame tracking is compared with random performance and remains inadequate.

  • 8.1 NATURAL LANGUAGE UNDERSTANDING: The NLU task combines dialogue-act prediction with word-level IOB tagging for slot values.
  • 8.1 NATURAL LANGUAGE UNDERSTANDING: The NLU architecture uses character-trigram embeddings before separate predictors for dialogue acts and slot types.
  • 8.1 NATURAL LANGUAGE UNDERSTANDING: The two NLU outputs are trained simultaneously with a modified loss that reduces the influence of correctly predicted O labels.
  • 8.1 NATURAL LANGUAGE UNDERSTANDING: NLU baseline performance is reported as mean and standard deviation over ten leave-one-user-out splits.
  • 8.2 FRAME TRACKING: The rule-based frame tracker uses dialogue acts without references and existing frame slot values to predict frame references and creations.
  • 8.2 FRAME TRACKING: Its rules create or switch frames when informed values conflict, switch by matching values, and assign references using matching or recency.
  • 8.2 FRAME TRACKING: The rule-based baseline performs only slightly better than random for frame identification and similarly for frame creation.

9 CONCLUSION AND FUTURE WORK

The paper introduces Frames as a memory-focused dialogue corpus and formalizes frame tracking, which simultaneously maintains multiple semantic frames. It concludes that baseline performance leaves substantial room for improvement and identifies future work on modeling and scalable annotation.

  • Frames is a corpus of human-human dialogues for researching memory in goal-oriented dialogue systems.
  • Frame tracking extends state tracking by maintaining several semantic frames simultaneously throughout a dialogue.
  • The baseline results show substantial room for improvement on frame tracking.
  • Frames can also support research on memory in dialogue management and natural-language information presentation.
  • Future work targets new frame-tracking models and methods for scaling data collection and annotation.

A DATABASE OVERVIEW

The database overview distinguishes searchable from non-searchable fields in the vacation-package database.

  • Table 7 lists the searchable fields in the package database.
  • Table 8 lists the non-searchable fields in the package database.

B DIALOGUE ACTS AND SLOT TYPES

This section provides reference tables for the dialogue acts annotated in Frames and the slot types absent from its database.

  • Table 9 lists the dialogue acts used in the annotation of Frames.
  • Together, the tables document annotation categories and database omissions in Frames.
  • Table 10 lists slot types that are not present in the database.
Loading 1704.00057v2…