ParlAI: A Dialog Research Software Platform

1 Introduction

• Dialog is a fundamental research goal because capable agents must achieve many machine-learning subgoals, some of which directly affect the task.The paper frames communication goals as typically involving two or more communicators.Source: §1 · p. 1
• Dialog spans thousands of related tasks with a shared input-output format, making task transfer useful across skills such as reasoning, planning, and learning from interaction.Examples range from booking restaurants and chatting to answering factual or perceptually grounded questions.Source: §1 · p. 1
• Working on isolated datasets can silo research and encourage methods that overfit dataset-specific qualities instead of generalizing across tasks.Examples include WebQuestions-specific knowledge-base methods, SQuAD methods that predict context indices (Rajpurkar et al., 2016), and bAbI methods relying on supporting facts or its simulated nature (Weston et al., 2015).Source: §1 · pp. 1–2
• ParlAI unifies training and testing dialog models across tasks, supports multitask training and evaluation, and integrates Amazon Mechanical Turk.Its first release supports over 20 tasks and includes neural-model examples using PyTorch and Lua Torch; Theano or TensorFlow can also be used.Source: §1 · p. 2
• The platform aims to give the community easy access to dialog tasks and learning algorithms to help advance the field.The paper details these goals and provides a technical overview of the platform.Source: §1 · p. 2

• ParlAI standardizes dataset input formats and evaluation frameworks and metrics, while letting researchers share tasks and training code to improve reproducibility and follow-on research.The framework aims to unify the formats fed to machine-learning agents and standardize evaluation as much as possible.Source: §2 · p. 2
• By combining real and simulated language datasets and making multitask models as easy to build as single-task ones, ParlAI encourages task transfer and may reduce dataset-specific overfitting.The paper identifies task transfer as an important prerequisite for a general dialog agent.Source: §2 · p. 2
• ParlAI supports collecting, training, and evaluating live human dialog through Amazon Mechanical Turk, enabling easier connections between Turkers and agents and comparison across research groups.The paper notes that comparing Turk experiments across groups has historically been difficult.Source: §2 · p. 2
• ParlAI aims to motivate new tasks and agents that advance a common general dialog model, with repository tasks building toward that goal rather than standing solely as independent research.Source: §2 · p. 2

• A shared API lets agents work across tasks or multiple tasks at once, spanning supervised and interactive learning with real-language and simulated tasks.Source: §3 · p. 2
• ParlAI supports image-and-text tasks such as visual question answering and visually grounded dialog, and automatically downloads datasets on first use.Source: §3 · p. 2
• Mechanical Turkers can be embedded in task environments to collect data, train agents, or evaluate them.Source: §3 · p. 2
• ParlAI supports PyTorch and Lua Torch, uses ZeroMQ to communicate with non-Python languages, and includes batch and hogwild training.Source: §3 · p. 2

4 Worlds, Agents and Teachers

• A world is the environment, while agents act in it and may be learners, hard-coded bots, or humans.Source: §4 · pp. 2–3
• A teacher is an agent that talks to the learner to teach it, for example by implementing a task.Source: §4 · p. 3
• After defining the world and its agents, the main loop calls world.parley() for one time step during training, testing, or displaying.Source: §4 · p. 3

5 Actions and Observations

• All agents use one observation/action dictionary to exchange text, labels, and rewards; `act()` returns it and `observe()` receives it.The same object type serves both speaking and listening, with different field values for each use.Source: §5 · p. 3
• Messages can carry text, speaker identity, receiver reward, episode-done status, expected labels, teacher-supplied label candidates, ranked learner predictions, and teacher-provided performance metrics.Label and candidate fields support supervised tasks; ranked text candidates enable ranking metrics rather than evaluation of only the single text response.Source: §5 · p. 3
• Optional image fields support visual tasks, and the extensible dictionary can accommodate audio, other sensory data, and non-speech actions.Fields are technically optional, although text is likely to appear in nearly all exchanges.Source: §5 · p. 3

6 Code Structure

• The five main directories are core, agents, examples, tasks, and mturk, separating platform code, models, utilities, task implementations, and Mechanical Turk tools.The mturk directory also contains sample Mechanical Turk tasks.Source: §6 · p. 3

6.1 Core

• Core files define Agent and Teacher classes, dialog worlds and training wrappers, language dictionaries, evaluation metrics, and command-line parameter parsing.Agents implement observe() and act(); MultiTaskTeacher supports multitask training, while metrics include exact match, F1, and ranking measures.Source: §6.1 · pp. 3–4

6.2 Agents

• The agents directory includes DrQA, end-to-end memory networks, GRU seq2seq, remote and IR-baseline agents, and a repeat-label debugging agent.DrQA is implemented in PyTorch, and the memory network in Lua Torch.Source: §6.2 · p. 4

6.3 Examples

• Example mains display task data or model predictions, evaluate models, and train models with logging and optional alternating validation.The utilities accept command-line task and model choices.Source: §6.3 · p. 4

6.4 Tasks

• ParlAI supports over 20 tasks across question answering, sentence completion, goal-oriented dialog, chit-chat, and visual dialog.The release includes datasets such as SQuAD, bAbI tasks, Ubuntu, and VQA.Source: §6.4 · p. 5
• Tasks share a dialog API, can be downloaded automatically on first use, and allow agents to train and test across datasets or multitask via comma-separated choices.Users can select multiple tasks or all tasks from the command line.Source: §6.4 · p. 5

6.5 Mechanical Turk

• Mechanical Turk users act as ParlAI agents through the shared observation/action interface, enabling human-human, human-bot, and group conversations without agent code changes.The platform includes tools for collecting question-answer pairs and gathering ratings of bots.Source: §6.5 · p. 5
• Mechanical Turk supports online training and evaluation with Turkers conversing with machine-learning agents, while shared task procedures can be checked into the repository for reproduction.New MTurk tasks are configured with a main file and task-specific parameters.Source: §6.5 · pp. 5–6

• The experiment combines single-task and multitask DrQA training on SQuAD and bAbI, a combination the authors say had not previously been shown.The authors present it as demonstrating how easily ParlAI supports such an experiment.Source: §7 · p. 6
• The dialog-only API omits SQuAD answer-span indices, yielding 66.4 EM versus 69.5 EM for Chen et al. (2017).Because the true answer location is unknown, training randomly selects start and end indices of a context phrase matching the answer.Source: §7 · p. 6
• The results illustrate the limits of methods specialized for context-span prediction, since some bAbI tasks ask yes/no questions whose answers are absent from the context.The authors argue this supports avoiding focus on a single dataset and encourage generally applicable algorithms that benefit from training across dialog datasets.Source: §7 · p. 6
• DrQA performs well on SQuAD but does not match the best bAbI methods, and multitasking does not help.The experiment reports test accuracy on bAbI 10k and SQuAD using Exact Match; the bAbI subset has answers exactly contained in the text.Source: Table 1 · p. 6, §7 · p. 6

8 Related Software

• ParlAI aims to unify independent dialog datasets and model training code that use differing formats and supervision.The paper describes this landscape as fragmented.Source: §8 · p. 6
• Gym and Universe6 support reinforcement-learning algorithm development, but neither focuses on dialog or supervised datasets.Gym covers games such as Pong or Go, while Universe covers online games and websites.Source: §8 · p. 6
• CommAI7 uses textual communication and increasingly complex synthetic tasks for artificial general intelligence, unlike ParlAI’s focus on real dialog.The paper characterizes CommAI as operating in a reinforcement-learning setting and distinguishes its goal from evaluating machine intelligence directly.Source: §8 · p. 6

• ParlAI lets the research community share existing and new dialog tasks and agents that learn on them.Source: §9 · p. 6
• ParlAI supports collecting and evaluating conversations between agents and humans via Mechanical Turk.Source: §9 · p. 6
• The authors hope ParlAI will enable systematic development and evaluation of dialog agents, advance the state of the art, and benefit the field.Source: §9 · p. 7