Source-linked AI summary
TEACh: Task-driven Embodied Agents that Chat
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, Dilek Hakkani-Tur
TL;DR
Robots need dialogue to understand instructions, resolve ambiguity, and recover from mistakes during household tasks. TEACh provides over 3,000 human–human interactive sessions with an oracle-informed Commander and embodied Follower, and introduces three benchmarks for dialogue understanding, grounding, and task execution. Initial results show that modeling both agents end-to-end is difficult, while future progress requires few-shot generalization to more complex tasks.
Problem
Existing embodied benchmarks often omit continued communication, motivating study of dialogue for object-centric, hierarchical household tasks.
Method
TEACh collects over 3,000 human–human task sessions and defines EDH, TfD, and TATC benchmarks for modeling Followers or both agents.
Results
Engineering rule-based solvers for end-to-end Two-Agent Task Completion is difficult, while initial Episodic Transformer models provide baselines for EDH and TfD.
Takeaways & Limitations
TEACh supports research on grounding language in visual environments and actions while coordinating long-term and intermediate household goals through dialogue.
Takeaways & Limitations
TEACh tasks are more complex than ALFRED’s simple tasks, making few-shot generalization to new tasks important for Follower agents.
Abstract
from arXiv · showhide
Robots operating in human spaces must be able to engage in natural language interaction with people, both understanding and executing instructions, and using conversation to resolve ambiguity and recover from mistakes. To study this, we introduce TEACh, a dataset of over 3,000 human--human, interactive dialogues to complete household tasks in simulation. A Commander with access to oracle information about a task communicates in natural language with a Follower. The Follower navigates through and interacts with the environment to complete tasks varying in complexity from "Make Coffee" to "Prepare Breakfast", asking questions and getting additional information from the Commander. We propose three benchmarks using TEACh to study embodied intelligence challenges, and we evaluate initial models' abilities in dialogue understanding, language grounding, and task execution.
1 Introduction
TEACh studies language grounding and dialogue in object-centric, hierarchical household tasks through over 3,000 human–human sessions. It introduces three benchmarks and initial models for dialogue-conditioned task execution.
- TEACh contains over 3,000 human–human sessions interleaving language and environment actions during household tasks.A Commander has oracle task and world knowledge, while a Follower interacts with the environment and communicates in written English.
- The dataset targets grounding natural language to visual environments and actions while handling long-term goals, intermediate goals, and dialogue.
- TEACh dialogues are unconstrained and include variation in instruction granularity, completeness, relevance, overlap, coreference, past actions, and locations.
- Human Followers make mistakes and receive language-guided correction, unlike planner-based sessions.
- TEACh provides three benchmarks—EDH, TfD, and TATC—for studying embodied intelligence, with initial model evaluations.EDH and TfD model the Follower, while TATC models both agents end-to-end.
2 Related Work
Prior embodied-language benchmarks commonly omit further communication, while related work shows benefits from clarification and conversational history. TEACh extends this setting to object-centric, hierarchical task completion.
- Many visual-language navigation benchmarks translate an initial instruction and visual observation into actions without further language communication.
- Vision-and-language task-completion research extends beyond navigation to object interactions, partial observability, and object-state changes.
- Some datasets use planner-generated demonstrations, whereas others collect human instructions and demonstrations separately.
- TEACh gathers human instructions and demonstrations simultaneously in conversational task-completion sessions.
- Related embodied-navigation studies use simulated clarification or human–human dialogue to improve navigation.
3 The TEACh Dataset
TEACh is a replayable AI2-THOR dataset of 3,047 successful human gameplay sessions, built around extensible hierarchical task definitions and language-mediated interaction. Its sessions cover diverse tasks, environments, actions, and dialogue patterns.
- TEACh collected 3,047 human–human gameplay sessions for household tasks in the AI2-THOR simulator.Sessions include initial states, Commander oracle-information actions, dialogue, Follower movement, object interactions, and final states.
- Household Tasks: The task definition language specifies object properties and evaluates whether simulator states satisfy task criteria.For example, making coffee succeeds when a mug is clean and filled with coffee.
- Household Tasks: TEACh task definitions support parameterized and hierarchical tasks, including abstract object classes, determiners, and composite tasks such as Prepare Breakfast.
- Gameplay Session Collection: The Commander receives task and environment information, while the Follower navigates and performs object manipulation through egocentric visual interaction.Follower actions include navigation and manipulation operations such as pickup, place, open, toggle, slice, and pour.
- TEACh Statistics: 3,047 successful sessions are replayable for model training, feature extraction, or evaluation and span all 30 AI2-THOR kitchens.The dataset also includes most of the simulator’s living rooms, bedrooms, and bathrooms.
- TEACh Statistics: Successful sessions contain over 45k utterances, averaging 8.40 Commander and 5.25 Follower utterances per session.Simple tasks require fewer dialogue acts and Follower actions than composite tasks such as Prepare Breakfast.
4 TEACh Benchmarks
TEACh derives three benchmarks from interactive sessions to evaluate embodied agents using dialogue, environment observations, and action histories. The benchmarks cover follower action prediction, trajectory prediction, and end-to-end Commander–Follower task completion, with evaluation based on resulting environment state changes.
- Benchmark overview: TEACh defines EDH, TfD, and TATC to evaluate embodied AI models completing household tasks through natural-language dialogue.EDH and TfD model the Follower, whereas TATC models both Commander and Follower agents end-to-end.
- Execution from Dialogue History (EDH): EDH instances condition Follower action prediction on dialogue history and evaluate whether inferred actions produce the expected state changes.Inference ends when the agent predicts Stop, reaches 1000 steps, or reaches 30 failed actions.
- Data construction and evaluation: The released EDH data filters instances so that only state changes directly resulting in task progress are considered.TEACh also uses validation and test splits in seen and unseen environments, distinguishing rooms present or absent during training.
- Trajectory from Dialogue (TfD): TfD asks a Follower model to infer the session’s environmental action sequence from dialogue, without observing dialogue actions during inference.The task tests long-horizon action prediction using a block of instructions, and evaluation compares final state changes with those between the initial and final session states.
- Task scope: TEACh tasks span simple and composite objectives, with composite tasks such as PREPARE SALAD containing subtasks and potentially requiring more actions.Per-session task statistics are reported as averages with standard deviation across task types.
- Two-Agent Task Completion (TATC): TATC gives both agents only environment observations, requiring the Commander to obtain task information and communicate it incrementally to the Follower.Success and goal-conditioned success compare the predicted end state with state changes between the initial and final states.
5 Experiments and Results
Experiments adapt the Episodic Transformer for dialogue-conditioned follower benchmarks and compare it with unimodal, random, and rule-based baselines. Results show meaningful gains on EDH but poor TfD performance and substantial difficulty engineering planner-style TATC agents.
- Experiments and Results: The experiments establish initial baseline performance and demonstrate the richness of TEACh data and the difficulty of its benchmarks.Table 4 reports success rate and goal condition success rate, while Table 5 summarizes rule-based-agent success across task types.
- Follower Models for EDH and TfD: The adapted Episodic Transformer uses dialogue utterances as language input and environment actions with visual observations as past-action history.It predicts the next action and target object using a TEACh-specific action head, while optionally applying history loss over the full action sequence.
- Follower Models for EDH and TfD: All E.T. conditions significantly outperform Random and Lang-Only on EDH success rate and goal condition success rate across splits.Against Vision-Only, improvements are significant on unseen but not seen splits.
- Follower Models for EDH and TfD: E.T. performs below its ALFRED success rates, reaching 38.24% on ALFRED test-seen and 8.57% on ALFRED test-unseen splits.ALFRED initialization shows a small but statistically insignificant benefit, while TEACh language includes multiple speakers, irrelevant utterances, and dialogue anaphora.
- Follower Models for EDH and TfD: E.T. performance on TfD is poor but non-zero, unlike Random, with instances averaging approximately 130 trajectory steps versus approximately 50 in ALFRED.The low initial performance prevented additional TfD ablations.
- Rule-based Agents for TATC: Rule-based TATC agents required about 150 engineering hours and still could not solve most compositional tasks.Their policies solved some simple tasks about half the time, while policies for several composite tasks were not successfully developed.
6 Conclusions and Future Work
The paper presents TEACh as a dataset and benchmark suite for situated dialogue, grounding, and household task completion. Future work emphasizes stronger generalization across complex tasks and human-in-the-loop evaluation of two-agent models.
- Conclusions: TEACh contains over 3,000 situated human-human dialogues for household tasks in AI2-THOR, with grounding, varied instruction granularity, and unconstrained speaker interleaving.The paper also introduces an extensible task-definition language and three benchmarks: EDH, TfD, and TATC.
- Future Work: Few-shot generalization to new tasks is identified as critical because TEACh requires learning several tasks more complex than ALFRED's simple tasks.The authors propose applying other ALFRED modeling approaches to EDH and TfD.
- Future Work: Human-in-the-loop evaluation is proposed for future Commander and Follower models developed for TATC.
A Additional TEACh Statistics
TEACh statistics describe task coverage, human completion difficulty, language and action distributions, and planned extensions to task and scene diversity. Human success was higher on simpler tasks and lowest on more complex or reasoning-intensive tasks.
- Dataset coverage: At least 50 sessions per task were targeted for unseen validation and test splits, but final counts varied with annotator success and simulator nondeterminism.The final number of sessions per task and split is reported in Table 6.
- Dataset coverage: Unseen splits reuse ALFRED floorplans, while future work plans entirely new floorplans and layouts through controllable generation.
- Human performance: Human success rates were higher for simpler tasks such as Make Coffee and Water Plant and lowest for Prepare Breakfast and Boil Potato.Prepare Breakfast had the most steps, while Boil Potato required additional reasoning about using a smaller container to fill a larger one.
- Dataset statistics: The dataset reports word, part-of-speech, object-interaction, dialogue-length, utterance-length, and Follower-action distributions across sessions.The distributions include log-scaled views for several statistics, and sessions can contain up to 139 utterances.
- Future benchmark scope: The extensible task language includes Prepare Study Desk and simpler tasks such as Turn On/Off All Lights for future TATC benchmark iterations.
B Data Cleaning
TEACh releases both original gameplay utterances and cleaned versions to reduce spelling and chat-normalization noise while preserving intended content.
- Cleaning procedure: Cleaned utterances remove spelling errors using SymSpell followed by manual checking to avoid spurious changes.
- Cleaning procedure: The cleaning process expands common chat contractions, including changing “nvm” to “never mind.”
- Cleaning procedure: Utterances referring only to annotation-interface aspects were removed from the cleaned data.
C Annotator Instructions
Annotator instructions establish complementary Commander and Follower roles: the Commander provides task information and dialogue support, while the Follower navigates and performs environment interactions. The protocol uses tutorials, priming, search assistance, incremental instructions, and troubleshooting guidance.
- Role setup: The Commander can inspect task progress and communicate but cannot pick up or place objects, whereas the Follower can manipulate objects but cannot see the progress check.Both players use chat to coordinate task completion.
- Coordination protocol: Followers are instructed to ask the Commander for object locations first and search independently when the Commander cannot locate an object.Searching includes opening drawers and cabinets.
- Coordination protocol: Commanders are instructed to use the progress-check search function and provide multi-step tasks incrementally rather than all at once.The examples require waiting for the Follower to finish one step before giving the next.
- Troubleshooting: Annotators are instructed to help resolve execution problems through dialogue, such as checking environmental conditions and removing obstructing objects.
- Role setup: Workers were randomly assigned to Commander or Follower roles after reading priming instructions and completing a two-player session setup.Only annotators who successfully completed the tutorial tasks were allowed to participate in the main annotation.
- Interfaces: The Commander interface provides task progress, object-location indications, Follower and Commander views, navigation controls, and chat, while the Follower interface provides egocentric and top-down views, chat, navigation, and object interaction controls.
D Additional EDH and TfD Experiment Details
Additional experiment details characterize EDH instance lengths and describe the E.T.-based modeling setup, training losses, computational costs, and task-level evaluation considerations. EDH instances can require long histories and many predicted actions, including substantial object interaction.
- EDH statistics: The average EDH action history is 86.97 actions, with a significant number of instances exceeding 200 actions.Dialogue and action history distributions are shown in Figures 13 and 14.
- EDH statistics: EDH models predict an average of 19.76 actions, including 4.74 object interaction actions, while some instances require up to 324 predicted actions.Many instances require over 50 predicted actions, and a significant number require 10–20 object interactions.
- Modeling setup: The E.T. setup replays human demonstrations to generate 900 × 900 image observations and resizes them for ResNet-50 and Mask R-CNN-based processing.The visual encoder and mask generator are pretrained on ALFRED expert demonstrations and remain frozen during training.
- Training objectives: E.T. uses equally weighted cross-entropy losses for action prediction and object-category prediction during object interactions.Auxiliary ALFRED progress losses are omitted because TEACh lacks equivalent subgoal progress signals.
- Computational cost: Episode replay, preprocessing, training, and EDH evaluation require approximately 6, 7, 5, and 6 hours respectively on specified EC2 GPU or thread configurations.
- Evaluation: EDH success rates are broken down by the original gameplay task, although hierarchical task definitions allow instances from different tasks to share predicted steps.Make Coffee can appear as a subtask within Prepare Breakfast sessions.
E Rule-Based Agents for TATC
The rule-based TATC agents use progress checks and handcrafted task logic to turn task state into language instructions and executable action sequences.
- The rule-based Commander and Follower agents were engineered as an attempt to solve the TATC benchmark.
- The Follower queues action tokens and asks “What should I do next?” when its queue is empty.The Commander responds by executing a Progress Check and generating a templated low-level instruction.
- The Commander uses Progress Check outputs containing task objectives and problem keys to determine supervision.Problem keys encode properties that must be satisfied to complete the task.
- For parentReceptacle goals, handcrafted logic navigates to the object and selects interaction steps based on its container state.The procedure can toggle or open a parent object, pick up the target, and advance an internal step state.
F Task Definition Language
TEACh defines household tasks through components, relations, properties, and parameters, enabling hierarchical and reusable task specifications checked against simulator state.
- The Task Definition Language specifies household tasks through object properties that must be satisfied for success.It is based on a PDDL-like syntax.
- A task combines components and relations, with components describing required object conditions and relations describing relationships between components.The current relation mechanism uses parentReceptacles to check containment.
- Parameters act as macros that are text-replaced before processing, enabling low-effort creation of task variants with different object classes or descriptions.Clean X can be instantiated with values such as Plate or Utensil.
- The task anchor object identifies which component resolves to the object checked in a relation.In Make a Plate of Toast, the toast anchor resolves to BreadSliced and the Clean X parameter resolves the plate anchor to Plate.
- Task components reference other tasks, allowing complex tasks such as Make a Plate of Toast to compose Toast and Clean X subtasks.The example requires toast to be placed on a clean plate.
- Atomic components use conditions, failure descriptions, and determiners to specify desired properties, annotator feedback, and required object counts.Determiners may be a, all, or a positive integer.
G TEACh Examples and Qualitative Analysis
TEACh examples show unconstrained human dialogue interleaved with actions, spanning instruction granularity, corrections, help-seeking, feedback, and imperfect coordination.
- TEACh was collected through unconstrained chat between a Commander holding task information and a Follower taking environment actions.The dataset contains 3,047 human–human gameplay sessions in AI2-THOR, with utterances interleaved with movement and object interactions.
- Commander guidance ranges from high-level requests to detailed step-by-step instructions, requiring agents to map different granularities to low-level actions.Make Coffee examples contrast a brief request with explicit sequential guidance.
- Dialogues include recovery from mistakes, including corrected object locations, changing placement instructions, and help when objects or containers cause problems.Examples include searching for a container, correcting a mug location, and resolving a pot that does not fit in a sink.
- Standard teacher forcing does not account for human instruction and execution imperfections, so robust models must distinguish essential information from irrelevant or incorrect guidance.Human Commanders may make mistakes and human Followers may not perfectly follow instructions.
- Without forced turn taking, messages can arrive out of order and instructions may overlap with unfinished actions.The examples include interleaved messages and Commanders providing subsequent steps before the Follower completes the previous one.
- Followers may request clarification, object locations, or assistance, while also confirming progress and anticipating steps before explicit instructions.These behaviors appear in examples involving tissue boxes, mugs, sliced objects, sandwiches, and plants.