Source-linked AI summary
ParlAI: A Dialog Research Software Platform
Alexander H. Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, Jason Weston
TL;DR
Dialog is a fundamental NLP goal, but methods can overfit individual datasets instead of generalizing across tasks. ParlAI addresses this with a unified framework for sharing, training, evaluating, and collecting dialog data; its demonstrative experiment shows both broad applicability challenges and the need for generally applicable learning algorithms.
Problem
Dialog is a fundamental NLP goal, while methods specialized to datasets such as WebQuestions, SQuAD, or bAbI may not generalize to other tasks.
Method
ParlAI provides a unified dialog API and platform for sharing tasks and agents, training and testing across multiple tasks, and integrating Mechanical Turk and interactive learning.
Results
ParlAI supports over 20 tasks in its first release, while DrQA performs well on SQuAD but does not match the best methods on bAbI and multitasking does not help.
Takeaways & Limitations
The platform standardizes task access and evaluation to support reproducibility and challenges the community to develop dialog algorithms that generalize across datasets.
Takeaways & Limitations
The demonstrative experiment randomly selects start and end indices for context phrases matching the training answer because the true answer location is unknown.
Abstract
from arXiv · showhide
We introduce ParlAI (pronounced "par-lay"), an open-source software platform for dialog research implemented in Python, available at http://parl.ai. Its goal is to provide a unified framework for sharing, training and testing of dialog models, integration of Amazon Mechanical Turk for data collection, human evaluation, and online/reinforcement learning; and a repository of machine learning models for comparing with others' models, and improving upon existing architectures. Over 20 tasks are supported in the first release, including popular datasets such as SQuAD, bAbI tasks, MCTest, WikiQA, QACNN, QADailyMail, CBT, bAbI Dialog, Ubuntu, OpenSubtitles and VQA. Several models are integrated, including neural models such as memory networks, seq2seq and attentive LSTMs.
1 Introduction
Dialog research spans many related tasks and skills, but dataset-specific work can produce siloed methods. ParlAI addresses this with a unified platform for training, testing, multitask research, and Mechanical Turk integration.
- Dialog covers tasks ranging from restaurant booking and sports or news chat to factual and perceptually grounded questions.
- Methods specialized to WebQuestions, SQuAD, or bAbI may fail to generalize because they exploit dataset-specific representations or assumptions.
- ParlAI provides a unified framework for training and testing dialog models, including multitask training or evaluation across many tasks.
- The first release supports over 20 tasks and integrates Amazon Mechanical Turk for dialog research workflows.
- The platform aims to provide community access to tasks and learning algorithms that perform well on them.
2 Goals
ParlAI’s goals are to unify dialog development, support diverse skills and real human interaction, and move research toward a common general dialog model.
- Unified framework for development of dialog models: ParlAI unifies dialog dataset formats and standardizes evaluation frameworks and metrics as much as possible.
- Unified framework for development of dialog models: Its repository supports sharing new tasks and agent training code to aid reproducibility and follow-on research.
- General dialog involving many different skills: ParlAI combines real and simulated language datasets while making multitask model development and evaluation as easy as single-task work.
- General dialog involving many different skills: This design should reduce dataset-specific overfitting and encourage task transfer, an important prerequisite for a general dialog agent.
- Real dialog with people: Mechanical Turk integration enables live human dialog and comparison of Turk experiments across research groups.
- Towards a common general dialog model: The platform aims for new tasks and agents to contribute toward a common general dialog model rather than remain independent research pieces.
3 General Properties of ParlAI
ParlAI presents tasks and agents through a common interface that supports supervised, interactive, real-language, simulated, and multimodal dialog research.
- All ParlAI tasks use a single API, allowing one agent to be applied to any task or multiple tasks at once.
- The framework supports fixed supervised datasets, interactive online or reinforcement-learning tasks, and both real-language and simulated tasks.
- ParlAI also supports images alongside text for visual question answering and visually grounded dialog.
- The first release includes PyTorch and Lua Torch training examples, with ZeroMQ enabling communication with non-Python languages.
- Batch and Hogwild training are built into the code, and an example training main is provided.
4 Worlds, Agents and Teachers
ParlAI organizes dialog environments around worlds, agents, and teachers, then runs training, testing, or display through a common interaction loop.
- A world is the environment, ranging from two-agent conversation to complex multi-agent interactive settings.
- Agents act in worlds and may be learned systems, hard-coded bots, or humans such as Mechanical Turk workers.
- Teachers are agents that talk to learners to implement tasks, while world.parley() runs one world time step.
- The main loop can be used for training, testing, or displaying data after the world and agents are defined.
5 Actions and Observations
ParlAI uses a common observation/action Python dictionary for communication between agents and environments. The extensible format carries dialogue content, supervision, rewards, metrics, and other media.
- Agents exchange a common observation/action object through act() and observe(), using the same format for speaking and listening.The object passes messages between agents and the environment.
- The message can carry speech text, speaker identity, rewards, and an episode-completion signal.
- Supervised tasks may add expected labels, response candidates, ranked predictions, and performance metrics.These fields support question answering, multiple-choice or ranking tasks, and evaluation.
- The dict is extensible to support images and additional sensory data or non-speech actions.
- Fields are technically optional by dataset, although text is likely to appear in nearly every exchange.
6 Code Structure
ParlAI organizes tasks, agents, examples, and Mechanical Turk functionality into a modular codebase. Standardized APIs and command-line tools support task addition, model training, evaluation, multitasking, and human interaction.
- Codebase organization: The codebase has core, agents, examples, tasks, and mturk directories separating platform, models, workflows, datasets, and human-task functionality.
- Core library: The core library defines agent and teacher abstractions, worlds for dialogue and training, language dictionaries, evaluation metrics, and parameter handling.
- Agent interface: All agents communicate through the observation/action dict, which carries text, identities, rewards, episode status, labels, and candidate responses.
- Agents: Available agents include attentive LSTM DrQA, end-to-end memory networks, GRU seq2seq, remote agents, information-retrieval baselines, and repeat-label agents.
- Examples and workflows: Examples provide commands for displaying data, training and evaluating models, and running bAbI and SQuAD together as multitask inputs.
- Tasks: The first release supports over 20 tasks across question answering, sentence completion, goal-oriented dialogue, and chit-chat, with standardized task files for additions.
- Mechanical Turk: Mechanical Turk integration supports data collection, model evaluation, and online interaction, while new tasks can be added for sharing and reproducibility.
7 Demonstrative Experiment
A DrQA experiment illustrates ParlAI’s ability to run single-task and multitask training across SQuAD and bAbI while exposing limitations of dataset-specialized methods. The unified dialog interface avoids answer-span labels but incurs a small SQuAD performance decrease.
- DrQA was evaluated with single-task and multitask training on SQuAD and bAbI, a combination not previously shown with any method to the authors’ knowledge.
- 66.4 EM vs. 69.5 EM: ParlAI’s dialog-only API produced lower SQuAD Exact Match than the cited comparison because start and end indices were unavailable.The bAbI subset used answers exactly contained in the text.
- DrQA solves some bAbI tasks and performs well on SQuAD but does not match the best-performing bAbI methods.
- Multitasking does not help DrQA in the reported experiment.
8 Related Software
ParlAI addresses fragmented dialog datasets and model code by specializing in unified dialog research, unlike broader reinforcement-learning platforms and synthetic-task frameworks.
- Existing dialog datasets and model-training code use varied formats and supervision, creating a fragmented landscape that ParlAI aims to unify.
- OpenAI Gym and Universe support reinforcement-learning algorithm development for games, online games, and websites rather than dialog research.They also do not cover supervised datasets.
- CommAI uses textual communication and synthetic incremental tasks to pursue artificial general intelligence, whereas ParlAI emphasizes real dialog.The frameworks therefore have different research focuses.
9 Conclusion and Outlook
ParlAI lets researchers share dialog tasks and learning agents while collecting and evaluating human-agent conversations through Mechanical Turk. The authors argue that this enables systematic development and evaluation of dialog agents and benefits the field.
- ParlAI supports sharing existing and new dialog tasks and agents, plus collecting and evaluating human-agent conversations through Mechanical Turk.
- This tool enables systematic development and evaluation of dialog agents and helps advance dialog research.
- The platform is intended to benefit the dialog research field as a whole.