PaperlayerView exact revision on arXiv

Source-linked AI summary

Imitating Interactive Intelligence

Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, Petko Georgiev, Aurelia Guy, Tim Harley, Felix Hill, Alden Hung, Zachary Kenton, Jessica Landon, Timothy Lillicrap, Kory Mathewson, Soňa Mokrá, Alistair Muldal, Adam Santoro, Nikolay Savinov, Vikrant Varma, Greg Wayne, Duncan Williams, Nathaniel Wong, Chen Yan, Rui Zhu

arXiv:2012.05672v2cs.LGcs.AIcs.MA

TL;DR

Useful interaction is difficult to define and evaluate, especially when an untrained agent rarely succeeds. The paper develops agents for reciprocal language interaction and evaluates their behavior through scripted tasks and other strategies; the agents generalize and often act in ways that suggest understanding, though imitation learning also reproduces human errors.

  • Problem

    Useful interaction is ambiguous, and human evaluation of an untrained agent can provide little discriminative feedback when successful behavior is rare.

    Source: §2.2 · pp. 3–4

  • Method

    The paper develops agents that interact with humans and one another through instructions and questions, using imitation learning and an auxiliary visual loss based on objects in view during training.

    Source: §1 · p. 2, §2.5.2 · p. 13, §4 · p. 38

  • Results

    Agents generalized and transferred knowledge to new tasks, and even when they failed instructions, their actions often seemed to reflect some understanding of them.

    Source: §5 · pp. 38–39

  • Takeaways & Limitations

    In a 3D virtual environment, the authors developed robust and effective strategies for evaluating agents on tasks without easily programmed success criteria.

    Source: §5 · p. 38

    Imitation learning reproduces human errors such as misspellings, clumsiness, and lapses of attention, so it alone does not distinguish skill from error.

    Source: §4 · p. 38

Abstract

from arXiv · show

A common vision from science fiction is that robots will one day inhabit our physical spaces, sense the world as we do, assist our physical labours, and communicate with us through natural language. Here we study how to design artificial agents that can interact naturally with humans using the simplification of a virtual environment. This setting nevertheless integrates a number of the central challenges of artificial intelligence (AI) research: complex visual perception and goal-directed physical control, grounded language comprehension and production, and multi-agent social interaction. To build agents that can robustly interact with humans, we would ideally train them while they interact with humans. However, this is presently impractical. Therefore, we approximate the role of the human with another learned agent, and use ideas from inverse reinforcement learning to reduce the disparities between human-human and agent-agent interactive behaviour. Rigorously evaluating our agents poses a great challenge, so we develop a variety of behavioural tests, including evaluation by humans who watch videos of agents or interact directly with them. These evaluations convincingly demonstrate that interactive training and auxiliary losses improve agent behaviour beyond what is achieved by supervised learning of actions alone. Further, we demonstrate that agent capabilities generalise beyond literal experiences in the dataset. Finally, we train evaluation models whose ratings of agents agree well with human judgement, thus permitting the evaluation of new agent models without additional effort. Taken together, our results in this virtual environment provide evidence that large-scale human behavioural imitation is a promising tool to create intelligent, interactive agents, and the challenge of reliably evaluating such agents is possible to surmount.

1 Introduction

The introduction frames human-like intelligence as inseparable from physical and social interaction. It proposes embodied agents that use language interactively, going beyond instruction following to also produce language.

1 Introduction

  • Human-like intelligence requires studying how people behave as physical agents and communicate in groups, not only how they think abstractly.The introduction links social and linguistic capacities to the shaping forces of large-group interactions.Source: §1 · p. 2
  • The research program aims to build embodied agents that perceive and manipulate the world, use language, and respond to general human requests.The envisioned interaction includes shared attention, gesture, demonstration, and dialogue.Source: §1 · p. 2
  • Unlike instruction-following work, these agents are designed to generate instructions and questions as well as follow instructions and answer questions.The program emphasizes interaction with both humans and other agents.Source: §1 · p. 2

2 Our Research Program

The research program uses a virtual Playroom to study interactive agents, developing human interaction data and multimodal agents alongside imitation-based behavioural priors, regularisation, and visual-language auxiliary losses. It combines these with inverse-reinforcement-learning-based training and setter–solver interaction to approximate human language-game behaviour.

2.1 The Virtual Environment

  • The Unity-based virtual Playroom avoids robotic-hardware difficulties while integrating perception, control, and language in randomized rooms that support spatial reasoning and partial observability.Its variable layouts and objects enable interactions involving reference ambiguity, containment, construction, support, and occlusion.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Learning to Interact

  • Because utility is subjective and untrained agents rarely succeed or visit expert states, human-feedback training is inefficient; the work instead builds a human-like behavioural prior through imitation.At a 10^-3 success probability, about 1,000 trials may be needed before an evaluator sees a successful trial; imitation learning leverages information in intelligent human behaviour.Source: §2.2 · pp. 3–5

2.3 Collecting Data for Imitation Learning

  • To shape the data distribution, the researchers used language games with four interaction types, focusing on question-and-answer and instruction-following prompts that humans expanded into varied instructions.The prompt set used 24 base prompts and up to 10 modifiers, while participants could choose their own wording and vocabulary.Source: §2.3 · p. 6
  • The dataset contains 610,608 setter-solver episodes, with 549,468 for training and 61,140 for validation, and 320,144 unique setter utterances.Episodes lasted up to two minutes and averaged 55 ± 25 seconds; solvers took 28 ± 18 seconds on average to complete tasks.Source: §2.3 · pp. 7–8

2.4.1 Action Representation; 2.4.2 Perception and Language; 2.4.3 Network Components

  • Agents combine visual and language inputs through a ResNet, multimodal transformer, and LSTM, then produce movement and language actions using an autoregressive policy.The action space includes continuous mouse-look, keyboard movement, and grabbing; agents perceive 96×72 RGB images and act 15 times per second.Source: §2.4.1 · p. 8, §2.4.3 · p. 10, §2.4.2 · p. 10

2.5 Learning; 2.5.1 Behavioural Cloning

  • Training combines imitation learning with supervised and unsupervised objectives; behavioural cloning learns action sequences from expert observation-action demonstrations without agent trial and error.Behavioural cloning can therefore avoid trial-and-error interaction, but cannot use self-directed experience to learn further.Source: §2.5 · p. 10, §2.5.1 · pp. 11–12

2.5.2 Auxiliary Learning and Regularisation

  • Because acting produces novel observations whose desired responses are not specified by demonstrations, auxiliary learning and regularisation can support generalisation to unseen sequences.The agent’s actions alter the world and its subsequent observations, making unfamiliar inputs an inevitable feature of interaction.Source: §2.5.2 · pp. 12–13
  • Language Matching aligns visual representations with solver instructions, while Object-in-View trains representations to encode visible objects and colours using simulator labels available only during training.Language Matching contrasts visual input and setter language from the same versus different episodes; Object-in-View contrasts descriptions of visible coloured objects with fictitious objects.Source: §2.5.2 · p. 13

2.5.3 Inverse Reinforcement Learning

  • BC errors can compound as an imitator reaches states outside its training data; IRL instead infers demonstrator rewards and trains policies to return to likely states (Ross et al., 2011; Finn et al., 2016).This can support recovery from unseen conditions, unlike BC, which may reach states where it has not been trained to respond.Source: §2.5.3 · p. 14
  • High-dimensional GAIL discriminators risk overfitting pixel coincidences, so the method uses RandAugment and Language Matching to encourage image invariance and vision-language alignment.Random cropping, rotation, translation, and shearing perturb images without changing their human-understood content; Language Matching is active on demonstrator, not imitator, observations.Source: §2.5.3 · pp. 16–17
  • On the “Put X on Bed” task, BG·A performed best; G·A failed to improve, B picked objects at random, and BG and B·A improved on B.BG·A combines behavioural cloning (BC), GAIL, and auxiliary losses; BG combines BC and GAIL, while B·A combines BC and auxiliary regularisation.Source: §2.5.3 · p. 18, Figure 8 · p. 18

2.5.4 Interactive Training

  • During interactive training, setter agents generate instructions and solver agents act on them; BC uses both roles’ demonstrations, while discriminator rewards train solvers to accomplish tasks.The setter and discriminator together approximate human instruction-giving and preference feedback.Source: §2.5.4 · p. 19
  • Setter replay warm-starts solver training by replaying human setter actions in half of solver-training episodes; it improves performance and training time but limits continued interaction and is impractical for robotics.The replayed setter actions come from the language-games dataset; the method is helpful but not crucial.Source: §2.5.4 · pp. 19–20

2.6 Evaluation

Because interaction capability is a central measure of agent fitness and is difficult to evaluate, the paper uses multiple evaluation methods. These methods generally agree in ranking agents, while comprehensive human-agent interaction remains the gold-standard test.

2.6 Evaluation

  • Interaction capability is the fitness criterion for agents, but evaluating interactions may be as difficult as evaluating intelligence.The authors argue that an oracle for evaluating any interaction would imply human-level AI.Source: §2.6 · p. 20
  • Simple scripted metrics and testing environments generally agree with rankings from freewheeling human interactive evaluation.The authors also report that learned evaluation models predict human ratings on held-out episodes or agents.Source: §2.6 · p. 20
  • The evaluation approach spans automated behavioural metrics, scripted probe tasks, human video ratings, and direct human-agent interaction.Machine-learning evaluation models trained on prior human evaluations are intended to predict ratings of held-out episodes or agents.Source: §2.6 · p. 20

3 Results

The results evaluate agent behaviour through scripted probes, human ratings, transfer tests, and automated metrics, examining the effects of training approaches and data. They also assess combinatorial generalisation and whether a human-trained evaluation model tracks agent performance and human success judgements.

3.1 Training and Simple Automated Metrics

  • Training losses and tracked heuristics helped monitor optimization, but did not directly establish human-rated performance; BC and language-match losses declined while the GAIL discriminator loss first rose, then fell.Object mention accuracy was only a rough heuristic because valid questions could mention objects absent from the room.Source: §3.1 · pp. 21–22

3.2 Scripted Probe Tasks

  • The six scripted instruction-following and question-answering probes gave interactively trained agents performance as good as or better than comparisons, but human success on these tasks was not perfect under the same conditions.Environment callbacks scored solver responses, offering a cheap approximation of human language games; humans’ judgements did not always match the programmed success criteria.Source: §3.2 · p. 23

3.3 Action Prediction Metrics

  • Interactive GAIL agents performed worse than BC-trained agents on validation action-prediction loss despite performing best on scripted probes, making BC validation an unreliable task-performance guide.The probe tasks tracked human-rated success more closely than BC validation performance.Source: §3.3 · p. 24, Figure 10 · p. 24

3.4 Automated Setter Metrics

  • Auxiliary losses improved setter metrics over BC, while GAIL was not a key driver; the authors attribute the gain mainly to better links between visual information and language.Metrics included object mention accuracy, utterance length and count, and 1-gram entropy; the human distributions were included for comparison in Table 2.Source: §3.4 · p. 24, Table 2 · p. 25

3.5 Agent Behaviour and Discriminator Reward Traces

  • In a counterfactual episode where the setter requested a nonexistent red robot instead of the visible white one, the GAIL reward model gave little reward, suggesting sensitivity to instruction-behaviour consistency.For the original white-robot request, reward was positive at points when the solver found and placed the correct object.Source: §3.5 · pp. 25–26, Figure 12 · p. 27

3.6 Observational human evaluation

  • In observationally human-rated solver episodes, BGR·A succeeded 57.02%±0.89 of the time, versus 20.12%±1.13 for BC alone; B·A and BG·A achieved intermediate rates of 37.28%±0.84 and 46.80%±0.88.BGR·A achieved joint setter-and-solver success in 39.58%±0.9 of episodes, with success requiring both roles to be rated successful by human raters.Source: §3.6 · p. 29

3.7 Interactive Human Evaluation

  • In live human-agent interactions, BGR·A succeeded 59.01%±1.06 of the time, and agent success-rate ordering and magnitudes closely matched observational evaluation.Success was 60.10%±1.32 for action and 57.25%±1.75 for question-answering tasks, compared with 57.02%±0.89 in observational evaluation.Source: §3.7 · pp. 30–31, Figure 14 · p. 30

3.8 Scaling & Transfer

  • With each doubling of dataset size, scripted-probe performance rose by roughly the same increment, with faster instruction-following gains for BG·A than B·A.Multitask training performed as well as or better than single-prompt training; for Position, multitask training with 1/8 of positional data outperformed training on all positional data alone, and orange-duck performance was similar to baseline despite excluding those examples from training.Source: §3.8 · pp. 31–32, Figure 15 · p. 32
  • Agents trained without orange ducks performed almost as well as fully trained agents on orange-duck Lift and Color probes.The probes show combinatorial generalisation: agents interacted with an “orange duck” despite never encountering that entity in training.Source: §3.8 · p. 33

3.9 Evaluation Models

  • Because humans labelled 93.27% of instruction-following episodes successful, the evaluation model used balanced accuracy to account for class imbalance.The model was trained exclusively on human instruction-following data and selected using a validation score computed on human and agent data.Source: §3.9 · p. 33
  • The language matching loss improved the evaluation model’s validation score by 3.38%.The model optimised a convex combination of evaluation and language matching losses.Source: §3.9 · p. 35
  • The evaluation model achieved 82.17% balanced accuracy on human data and 62.47% on BGR·A rollouts, while additional human ratings averaged 90.24%.The model distinguished better- from worse-performing agents, though it was imperfect on agent rollouts.Source: Figure 16 · p. 34, §3.9 · p. 35
  • Predicted and actual success rates for agents correlated with an R2 of 0.923, with a trend line close to unity.The evaluation model was trained on human data alone, so its performance on agent data required out-of-distribution generalisation.Source: Figure 16 · p. 34, §3.9 · p. 35
  • A variant that also predicted when success occurred achieved a 75.84% validation score, compared with 76.08% for the transformer model.Humans annotated the time at which success was achieved.Source: §3.9 · p. 35

4 Discussion & Related Work

The discussion frames integrated AI as the study of perception, embodied control, and language together, and situates the work among research on interaction and imitation. It also identifies limits in existing data, evaluation, and imitation-based approaches, while treating the virtual environment as a starting point for later real-world work.

Integrated AI Research

  • Integrated AI research brings perception, embodied control, and natural language together, echoing earlier visions of interactive systems and learning through imitation.The discussion invokes Turing’s imitation game, Winograd’s SHRDLU, and human imitation as context for this integrated agenda.Source: §4 · pp. 35, 36, 38

Imitating Human Behaviour at Scale

  • Large-scale behavioural datasets remain difficult to apply to embodied agents: text omits motor and perceptual experience, and child recordings have not yet yielded behavioural models.The discussion also notes that systems for transferring human-intention understanding across embodiments, environments, and perspectives have not yet been developed.Source: §4 · pp. 36–37
  • Computer games offer interactive behavioural data, but their creator-defined win-or-lose objectives do not resolve how to evaluate ambiguous human interactions.The work presents strategies for addressing the absence of an unambiguous metric for human behaviour as a key contribution.Source: §4 · p. 37

Language, Interactions, and Robotics

  • Existing embodied-language work often uses short observation windows or constrained environments, while these agents also produce context-specific language.The discussion contrasts Playroom interactions lasting over a minute with prior work’s few-second windows and notes practical challenges in aligning language with perception and sparse output data.Source: §4 · p. 37
  • The virtual environment enables faster iteration with fewer hardware challenges than a typical robotics platform, and the authors identify real-world application as a next step.They describe simulation as a place to study a broader process for building intelligent agents.Source: §4 · p. 37

Challenges of the Approach

  • Imitation learning can bootstrap competence, but it also reproduces human errors and does not distinguish skill from counterproductive behaviour.The discussion calls for additional techniques, including learning from human evaluative feedback, and suggests that broader capabilities may require direct human interaction, new methods, or larger-scale data.Source: §4 · p. 38

5 Conclusion

The agents combine perception, manipulation, language, and interaction, and show transfer to new tasks. Their imperfect but sometimes instruction-aligned behaviour is presented as a basis for continued interaction and feedback.

5 Conclusion

  • The agents perceive and manipulate their environment, produce language, respond to general human requests, and transfer knowledge to new tasks.The paper also reports that the agents interact with the world, each other, and people.Source: §5 · p. 38
  • Even when instructions were not satisfied, agents often took actions that seemed to reflect some understanding and could benefit from interactive feedback.The authors note that agent behaviours were not perfect.Source: §5 · pp. 38–39
  • The authors suggest that agents capable of sustaining people’s interest may receive continued interaction, which could select for increasingly intelligent and useful agents.They describe this as a promising virtuous circle, not an established outcome.Source: §5 · p. 39

6 Authors & Contributions

The contribution roster credits a large team across agent development, imitation learning, data and tasks, environment development, evaluation, engineering, experiments, and writing.

6 Authors & Contributions

  • Abramson, Ahuja, Georgiev, Guy, Hill, Hung, and others contributed across agent development, imitation learning, data and tasks, experiments, engineering, and writing.Abramson is identified as technical lead; Georgiev, Hill, and Hung led sub-efforts in agent development, environment development, and imitation learning, respectively.Source: §6 · p. 39
  • Brussee, Cassin, Chhaparia, Clark, Damoc, and Wong are among contributors credited with environment development, data, or engineering.Wong and Hill are identified as sub-effort leads for environment development.Source: §6 · pp. 39–40
  • Kenton, Landon, Muldal, Savinov, and Varma contributed to evaluation-model development or its experiments, while Lillicrap and Wayne are listed as effort leads.The roster also credits Landon and Muldal with writing, and Lillicrap and Wayne with agent development and imitation learning.Source: §6 · p. 40

1 Playroom Environment Description

The Playroom is a configurable Unity environment with rooms and contents randomized by episode. Its varied layouts and object placements support embodied interaction in changing spatial configurations.

1 Playroom Environment Description

  • The Playroom is a configurable Unity room whose features are randomized across episodes.Its repository supplies small objects and furniture, and object colours are applied at random from the listed colours.Source: §1 · p. 47, Table 3 · p. 47

1.1 Objects and furniture in the Playroom

  • Each episode samples toys and furniture from the repository, with object colours chosen randomly from the listed set.The repository’s objects and colours are illustrated in Figure 17.Source: Table 3 · p. 47, §1.1 · p. 48, Figure 17 · p. 48

1.2 Randomisation

  • Room geometry varies within an L-shaped layout: its two longest walls range from 6 to 10 metres, and no part is narrower than 4 metres.The agent’s initial position and orientation also vary within the room.Source: §1.2 · p. 48
  • Episodes vary shelf count and placement, door and window positions, and furniture locations, with 2 to 4 furniture items placed against the walls.Shelf count ranges from 0 to 8.Source: §1.2 · p. 48
  • Small objects vary in position and orientation, with 2 to 6 on the floor and 2 to 6 on furniture.Floor-object quantity is sampled uniformly.Source: §1.2 · p. 48

2 Data

The data comprise human language-game demonstrations and annotations of prerecorded human- and agent-generated episodes. Collection procedures varied partners and roles, while annotation rules converted task-specific episode evidence into success labels.

2.2 Language Games

  • Language games paired a setter who instructed a solver in the Playroom, with keyboard-and-mouse control and chat communication.Setters received randomly sampled prompts and modifiers; episodes ended when the setter judged the task complete or infeasible, or after two minutes.Source: §2.2 · p. 49, §2.2.2 · pp. 49, 51

2.1 Human Participants

  • Participants were paired across two groups, switched setter and solver roles every 30 minutes, and were reshuffled hourly to interact with multiple partners.Participants were recruited through Google’s internal labeling platform, gave consent under DeepMind’s HuBREC protocol, and received a fixed hourly rate.Source: §2.2.1 · p. 49, §2.1 · p. 49

2.2.2 Detailed Instructions

  • Including modifiers increased language diversity and encouraged setters to describe object attributes such as colour or relative position, beyond object names.Some episodes omitted modifiers; Tables 4–6 list prompts, modifiers, and demonstration counts by combination.Source: §2.2.2 · p. 51, Table 6 · p. 51, Table 4 · p. 54

2.3.1 Annotation Interface

  • Annotations used binary reward sketches: raters marked whether and when an episode succeeded, and sketches were reduced to a single success label.Continuous graded sketches were considered time-consuming and difficult to apply consistently across prompts and raters.Source: §2.3.1 · p. 55, Figure 20 · p. 56

2.3.2 Generating Episodes for Annotation

  • Annotations covered human-human episodes and four agent-generated episode types, including agent interactions and solver episodes with live or replayed human setters.When setters were replayed or agent-controlled, episodes ran for the full two minutes because manual termination was unavailable.Source: §2.3.2 · p. 57, Table 7 · p. 57

2.3.3 Truncation of Frame Sequences for Annotation

  • Frame sequences were truncated to focus judgments: solver clips began at the setter’s first emission and allowed five seconds after the solver’s first emission, while setter clips ended at their first emission.Solver clips were capped at 60 seconds and setter clips at 75 seconds; these limits covered over 95% of the respective human episodes or emissions. Human annotation agreement is reported in Table 8.Source: §2.3.3 · pp. 57–58, Table 8 · p. 58

3 Agent Architecture

The agents combine first-person visual input, language, and task-specific features through a multimodal transformer and recurrent memory. Separate policy heads produce language and physical actions, while no-op policies control when actions reach the environment.

3.1 Inputs

  • Agents received first-person RGB images, environment prompts or empty text, and dialogue from the other agent, alongside auxiliary or hard-coded features.Images were 96 × 72 × 3, and miscellaneous inputs included previous language actions and elapsed-step features for no-op decisions.Source: §3.1 · p. 58, §3.1.1 · p. 59, §3.1.3 · p. 61

3.1.1 Perception

  • Figure 21’s heat-map pixels report each agent’s mean success rate per prompt as a fraction of human-baseline performance for that script.The figure summarizes observational human evaluation results.Source: Figure 21 · p. 59
  • A 25-convolution ResNet encoded images into 30 spatial feature vectors, each projected to 512 dimensions; text was tokenized, lowercased, typo-corrected, and embedded.The vocabulary used the 500 most frequent dataset words plus words found in procedural evaluation instructions, and text sequences were padded or truncated to 16 tokens.Source: §3.1.1 · p. 60, §3.1.2 · pp. 60–61

3.2 Sensory Integration by the Multi-Modal Transformer (MMT)

  • An eight-layer, eight-head multimodal transformer integrated 63 visual, text, and previous-language vectors with two learned CLS-like vectors, then passed aggregated features to an LSTM.The two CLS-like outputs and a mean-pooled embedding formed a 1536-dimensional vector for the recurrent memory.Source: §3.2 · p. 61

3.3 Memory

  • A two-layer, 512-dimensional LSTM provided memory, with its output concatenated to its input through a skip connection before policy heads.The resulting vector was 1024-dimensional.Source: §3.3 · p. 62

3.4 Outputs

  • Language and motor policy heads generated word emissions and movement, grab, and look actions, while no-op policies vetoed actions at each timestep.The language policy shared embeddings for encoding input and decoding output; the look policy refined gaze through five discrete steps, capped at 0.01 resolution.Source: §3.4 · p. 62, §3.4.1 · p. 62, §3.4.2 · p. 63

4 Agent Training

Training combines behavioural cloning with auxiliary objectives and GAIL-based reinforcement learning. The approach addresses sparse language supervision, trains agents in interactive settings, and uses setter replay to make early solver training more useful.

4.2 Supervised Learning (Behavioural Cloning)

  • Behavioural cloning maximized expert-action likelihood across policy heads, with language loss weighted 50 and movement loss weighted 1; language targets were spread across timesteps as single-word predictions.This smearing slightly shifted emission timing but performed better than predicting a full utterance at once; the authors note the result may be context-dependent.Source: §4.2 · p. 64

4.3 Unsupervised and Auxiliary Supervised Learning

  • Language targets occurred at roughly one timestep per 2,000 before smearing, and naive supervised training produced poor language policies despite extended training.Language data also grounded object words in visual input, supporting both setter language and object manipulation; auxiliary and unsupervised objectives supplied dense, discriminative gradients.Source: §4.3 · p. 65

4.3.1 Language Matching (LM)

  • Language Matching trained a classifier to distinguish real visual-instruction pairs from pairs with language shuffled across trajectories, and was used for solver behavioural cloning.The task exploited correlations between expert visual observations and setter instructions; it was inactive during interactive training and setter training.Source: §4.3.1 · pp. 65–66

4.3.2 Object-in-View Auxiliary Supervised Learning (OV)

  • Object-in-View trained agents to judge whether a proposed colour-object combination was visible, addressing setter difficulty recognizing distant objects in crowded scenes.The objective aimed to support object and colour identification across changes in position, angle, and partial occlusion.Source: §4.3.2 · p. 66

4.4 GAIL and Interactive Training

  • GAIL trained a discriminator to distinguish expert trajectories from agent trajectories and converted its output into rewards for reinforcement learning.The discriminator used expert sequences as positive examples and interactive or setter-replay sequences as negative examples; its history covered about one second, and language matching also trained its representations.Source: §4.4 · p. 67, §4.4.5 · p. 68, §4.4.8 · p. 69

4.4.2 Interactive Training

  • Interactive training ran one model as setter and solver, but reinforcement learning updated the policy only on solver trajectories; action repetition at 15 frames per second improved GAIL training.Each sampled action was repeated for two time steps, using every other observation and action from the original 30-frame-per-second data.Source: §4.4.2 · p. 67, §4.4.1 · p. 67

4.4.3 Setter Replay (SR)

  • Setter replay replaced the agent setter with a recorded human setter in half of early training episodes, reducing unhelpful solver interactions from invalid or infeasible agent instructions.The replay used the recorded room configuration and human setter actions step by step.Source: §4.4.3 · pp. 67–68

4.4.10 Reinforcement Learning

  • The policy and reward model were trained with distributed actor-learner reinforcement learning, combining supervised dataset trajectories with online solver trajectories and GAIL rewards.The discriminator processed multimodal observations and buffered temporal features; policy and language heads shared rewards and a value-function baseline.Source: §4.4.6 · p. 68, §4.4.7 · p. 68, §4.4.8 · pp. 68–69

5 Distributed Training Infrastructure

The distributed setup separates data replay, interactive experience generation, learning, and evaluation across actor and learner roles. Separate streams support supervised objectives, reinforcement learning, reward-model training, and scripted or dataset-based evaluation.

5 Distributed Training Infrastructure

  • Actors generated data on CPUs while learners updated the agent and reward model on TPUs, with parameters synchronized through cacher workers.The setup followed IMPALA; actor types differed in whether they replayed human data, generated interactive episodes, or replayed human setter behavior.Source: §5 · pp. 69–70, §5.2 · p. 72

5.1 Actors

  • Dataset actors teacher-forced human trajectories, interactive actors generated setter-solver episodes and solver rewards, and setter-replay actors combined stored rooms with human setter actions.The experiment used 2,000 dataset actors and 2,000 online actors, with online actors allocated to interactive training, setter replay, or both.Source: §5.1 · p. 71

5.2 Learners

  • The agent learner combined dataset batches for behavioural cloning and auxiliary losses with online solver batches for reinforcement learning, while the reward-model learner trained on expert and online data.Dataset trajectories also supported evaluation of action likelihoods and language metrics; separate online evaluation actors ran scripted probe tasks.Source: §5.2 · pp. 71–72, §5.3 · p. 72

6 Evaluation models

The evaluation models were designed to automate assessment of solver success from video and language, complementing human evaluations and supporting model tuning and reinforcement-learning rewards.

6 Evaluation models

  • The evaluation models predict binary episode success or failure from human annotations, which agreed with one another about 85–90% of the time.Episode labels used majority votes across annotators.Source: §6 · p. 73, §6.1.1 · p. 73

6.1 Architecture

  • The architecture processes observation trajectories and connects their modalities to judge success for an episode.The evaluation models were trained to predict human annotator labels, using decimated observation sequences.Source: §6.1 · p. 73, §6 · p. 73

6.1.1 Inputs

  • Inputs included sampled video frames and the first setter and solver utterances, with text typo-corrected and padded to fixed lengths.The model also used binary success labels; success-frame prediction used a separate sequence target.Source: §6.1.1 · p. 73

6.1.2 Models

  • The ResNet model embeds video, setter instruction, and solver emission, then combines modalities by concatenation or elementwise product.A separate transformer architecture also combined the three modalities with learnable segment and position embeddings.Source: §6.1.2 · p. 74

6.1.3 Losses

  • Auxiliary language-matching and success-frame losses were added to the supervised objective to improve representations and generalization to unseen episodes.Success-frame prediction was intended to address the different timing of success in human and agent episodes and was used only with the ResNet model.Source: §6.1.3 · p. 75

6.2 Model Selection

  • Models and hyperparameters were selected using a validation score computed on weak and strong previously trained agents.Selection also included the success threshold and training step, using smoothed online validation scores.Source: §6.2 · p. 76

6.3 Additional Instruction-Following Results

  • The success-frame loss with TSM yielded a model almost as good as the transformer, with a slightly lower validation score but higher balanced accuracy for some test agents.Combining the transformer architecture with success prediction was suggested as a possibility but was not tested.Source: §6.3 · p. 77

6.4 Question-Answering Results

  • Question-answering evaluation produced agent rankings close to correct, but balanced accuracy and correlation were significantly worse than for instruction-following models, with success rates likely overpredicted.The authors attribute the overprediction in part to few negative human training examples.Source: §6.4 · p. 79

7 Automated Evaluation Metrics

The section introduces automated measures for setter language and solver performance, alongside scripted probe tasks with objective success criteria.

7.1 Setter Language Metrics

  • Setter object error rate checks whether sentences name an object type present in the room, while colour object error rate checks valid object mentions with the true colour.Colour vocabulary was canonicalized, such as mapping “turquoise” to “blue”; these metrics do not detect errors in spatial relationships between objects.Source: §7.1 · p. 79

7.2.1 Automated Metrics

  • Solver metrics compare an agent’s trajectory with a human demonstration, including whether both first lifted the same object and whether they lifted objects of matching colour and type.The first-object-lifted metric is useful for relative comparisons across agents and during training, since instructions may permit multiple objects or require no lifting.Source: §7.2.1 · p. 80

7.2.2 Scripted Probe Tasks

  • Scripted probes use randomized episodes and explicit reward criteria to assess navigation, lifting, object positioning, and questions about colour, existence, or counts.For count probes, answer frequencies are uneven, so baseline controls are important for interpreting scores; agent performance was reported in Table 11.Source: §7.2.2 · pp. 80–81

8 Scaling Experiments

The experiments examine how scripted-probe performance changes with training-data scale for BC and GAIL agents.

8 Scaling Experiments

  • The scaling comparison used B·A and BG·A agents trained on randomly subsampled datasets ranging from 34K to 548K episodes.The tested dataset sizes were 548K, 274K, 137K, 68K, and 34K episodes.Source: §8 · pp. 81–82
  • Performance increased smoothly with training data for both BC and GAIL, while GAIL’s rate of increase was higher, particularly on instruction-following probes.The reported reward averages instruction-following and question-answering probe tasks.Source: §8 · p. 82

9 Transfer Experiments

Transfer experiments test whether training across tasks improves probe performance, reduces the data needed for a new task, and supports generalization to unseen colour-object combinations.

9 Transfer Experiments

  • Transfer experiments used a smaller network than the default setup and reran relevant baselines to keep comparisons fair.The network used four rather than eight transformer layers and 256 rather than 512 embedding dimensions; BG·A was used throughout these experiments.Source: §9 · p. 82

9.1.1 Single-task versus Multi-task

  • Training on all task data yielded higher scripted-probe rewards than training on a single task.The authors suggest that navigation, grasping, and object-name representations may transfer across tasks.Source: §9.1.1 · p. 83

9.1.2 Multi-task Data Efficiency

  • With only one-eighth of the positioning data, agents could learn the task when also trained on background data from other tasks; multi-task training required less positioning data for good probe scores.The experiment incrementally added fractions of a 73K-episode positioning dataset to background tasks.Source: §9.1.2 · p. 83

9.2 Colour-Object Generalisation

  • The held-out-object probes included distractors, such as a non-orange duck when testing instructions about orange ducks.This design checked whether agents used the colour-object combination rather than responding to object type alone.Source: §9.2 · p. 84
  • Agents successfully handled held-out colour-object combinations with only a small performance drop relative to an equal-size unrestricted-data control.Orange ducks were held out from both training data and the environment, while the object type and colour appeared separately in other examples.Source: §9.2 · pp. 83–84

10 Vocabulary and Spelling Correction Table

This section documents the agent’s vocabulary and typo-correction table, which maps spelling and token variants—including plural and numeric forms—to normalized words or phrases, sometimes to an empty string.

10.1 Vocabulary

  • The vocabulary covers common language, scene objects, spatial relations, colors, and descriptions of objects’ shapes and sizes.Examples include “chair,” “behind,” “green,” “circular,” and “larger.”Source: §10.1 · pp. 84–85

10.2 Typo Table

  • The custom typo dictionary maps digit strings and misspellings to normalized number words, object terms, and phrases.Examples include ’03’ to ’three’, ’0bjects’ to ’object’, and ’abed’ to ’a bed’.Source: §10.2 · p. 86
  • The dictionary also normalizes spelling variants and plural forms, while future work will explore subword tokenization to better construct and learn large vocabularies.For example, ’aircrafts’ maps to ’airplane’ and ’arrangements’ to ’arrangement’.Source: §10.2 · p. 86
  • The table maps misspellings and variants to canonical words, such as “robort” to “robot,” “shelfs” to “shelf,” and “wimdow” to “window.”Examples span object names, colors, and other vocabulary.Source: §10.2 · pp. 93, 95
  • It also normalizes concatenated forms into multiword phrases, including “theball” to “the ball” and “roomyesno” to “room yes no.”Other entries similarly split forms such as “toyrobot” into “toy robot.”Source: §10.2 · pp. 93–95
Loading 2012.05672v2…