Source-linked AI summary
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox
TL;DR
Household instruction following requires grounding language and egocentric vision in long, compositional action sequences with object interactions and state changes. ALFRED introduces a large interactive benchmark of expert demonstrations and language directives, and a sequence-to-sequence baseline achieves poor overall performance, exposing the difficulty of long-horizon task planning.
Problem
Existing language-driven benchmarks largely omit the object interactions, state changes, and task-oriented behaviors required for household instruction following.
Method
ALFRED provides planner-generated expert demonstrations with high- and low-level language instructions in interactive, partially observable AI2-THOR scenes.
Results
The sequence-to-sequence baseline achieves less than 5% success rates on ALFRED’s complex tasks.
Takeaways & Limitations
ALFRED captures challenges closer to language-driven robotics, including visual grounding, interaction masks, long horizons, and compositional planning.
Takeaways & Limitations
Models take more steps than experts and generalize poorly to unseen scenes because of visual complexity and failures to track state changes.
Abstract
from arXiv · showhide
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with non-reversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like "Rinse off a mug and place it in the coffee maker." and low-level language instructions like "Walk to the coffee maker on the right." ALFRED tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets. We show that a baseline model based on recent embodied vision-and-language tasks performs poorly on ALFRED, suggesting that there is significant room for developing innovative grounded visual language understanding models with this benchmark.
1. Introduction
ALFRED benchmarks grounding natural language in visually observed household actions, addressing task-oriented behaviors that existing language-action datasets largely omit. It combines long-horizon, compositional interactions with object state changes and evaluates a baseline that performs poorly on these challenges.
- ALFRED targets object interactions, state tracking, and references to previous instructions in household tasks.Its directives can combine goals such as rinsing and placing an object with navigation instructions grounded in visual context.
- ALFRED connects human language to actions, behaviors, and objects in interactive visual environments using high- and low-level instructions.The benchmark uses planner-based expert demonstrations in 120 indoor AI2-THOR scenes with partial observability and irreversible actions.
- 25,743 directives describe 8,055 expert demonstrations averaging 50 steps and producing 428,322 image-action pairs.Agents also predict pixelwise interaction masks for target objects rather than only object classes.
- ALFRED includes high-level goals and low-level natural language instructions for object and environment interactions.The dataset comparison contrasts these capabilities with datasets requiring less interactive action selection.
- Less than 5% success rates were achieved by a sequence-to-sequence baseline on ALFRED’s complex tasks.Although isolated subgoals were easier, the model lacked reasoning capacity for long-horizon and compositional planning.
- ALFRED is intended to narrow the gap between simulated language-driven task completion and real-world robot action.The benchmark captures challenges involving visually and physically realistic interactions.
2. Related Work
Prior benchmarks cover navigation, video descriptions, language-action mapping, or embodied question answering, but often omit interactive object-centered task execution. ALFRED instead combines natural language with navigation and manipulation in visually complex, partially observable environments.
- Vision-and-Language Navigation: Vision-and-language navigation datasets describe routes to locations through egocentric visual observations, while ALFRED requires completing object-centered tasks.Navigation research has improved substantially, but route following alone does not capture household manipulation.
- Embodied Task Benchmarks: ALFRED uses richer, partially observable scenes and seven manipulation actions, including picking up, turning on, and opening objects.It also represents state changes such as clean versus dirty and supports varied language and visual complexity.
- Visual Semantic Planning: Unlike earlier AI2-THOR planning with templated language and discrete object classes, ALFRED uses human instructions and class-agnostic pixelwise interaction masks.This requires grounding object selections spatially rather than choosing from predefined object categories.
- Instruction Following: Language-action instruction-following work generally lacks visual, interactive environments, whereas ALFRED requires navigation and object interaction to complete tasks.Embodied question-answering datasets typically use templated language or static scenes instead.
- Robotics Instruction Following: ALFRED spans 120 scenes, diverse object classes, and unseen test environments, unlike robotics instruction-following studies often limited to few scenes or objects.Its evaluation explicitly includes generalization to environments distinct from training.
3. The ALFRED Dataset
ALFRED is built from replayable planner-generated demonstrations paired with crowdsourced, validated language directives in interactive AI2-THOR scenes. Its task, scene, object, state, and split design supports evaluation under partial observability and environmental generalization.
- ALFRED contains 25,743 directives paired with 8,055 expert demonstration episodes, each combining a high-level goal with step-by-step instructions.Every demonstration can be deterministically replayed in AI2-THOR 2.0.
- Task Design: The seven task types are parameterized by object focus, destination receptacle, scene, and sometimes a base object.Demonstrations use combinations of object and receptacle classes across 120 indoor scenes.
- Expert Demonstrations: 2,685 task-parameter combinations receive three expert demonstrations each, yielding 8,055 demonstrations averaging 50 action steps.Randomized starting locations provide multiple demonstrations for the same parameter set.
- Expert Demonstrations: Planner-generated demonstrations encode agent and object states and environment dynamics in PDDL, with task-specific goal conditions.The planner has full observability, but trained agents receive only egocentric visual observations during action execution.
- Data Splits: Validation and test demonstrations are distinct from training, with seen folds reusing training scenes and unseen folds using separate scenes.The split measures generalization to new spaces and novel object-class variations.
- Language Directives: Validation removes directives judged to describe different actions, including incorrect object references or directions.At least two annotators validate the three directives associated with each demonstration, with a possible tie-breaker.
4. Baseline Models
The baseline is a CNN-LSTM sequence-to-sequence agent that attends to language while combining visual observations and previous actions to predict household-task actions and object masks. Progress-monitoring losses add temporal and subgoal tracking, but training with expert trajectories is difficult to replace with student forcing because some actions are irreversible.
- 4.1. Sequence-to-Sequence Models: The model encodes visual observations with a CNN and language instructions with a bidirectional LSTM before decoding low-level actions with an LSTM.The language input combines the high-level goal and step-by-step instructions, separated by a <SEP> token.
- 4.1. Sequence-to-Sequence Models: DAgger-style student forcing is non-trivial because obtaining demonstrations requires replanning, and some irreversible actions cannot be recovered from.For example, slicing the only apple in a task can make completion impossible if the student takes that action at the wrong time.
- 4.1. Sequence-to-Sequence Models: The decoder predicts both the next action and a pixelwise binary interaction mask, with object selection determined from the predicted mask and simulator ground-truth mask overlap.The action space is discretized into navigation, interaction, and Stop actions; action and mask predictions use separate networks and supervised losses.
- 4.1. Sequence-to-Sequence Models: At each timestep, the decoder attends to instruction tokens and combines the attended language feature, current visual feature, and previous action to update its hidden state.Attention weights are conditioned on the previous decoder hidden state, and the attended feature is a weighted sum of the encoded words.
- 4.2. Progress Monitors: Two auxiliary losses form SEQ2SEQ+PM by predicting normalized elapsed time and the normalized number of completed subgoals.The progress target is t/T, while the subgoal target is c_t/C; both are trained with L2 loss.
- 4.2. Progress Monitors: Progress monitoring helps distinguish visually similar states and encourages coarse tracking of progress through the language directive.Subgoals correspond to planned action segments such as navigation, pickup, and heating, each associated with a language instruction whose alignment must be learned.
5. Experiments
ALFRED evaluates agents on full household tasks and partial goal-condition completion using success and path-weighted metrics. The evaluation compares model executions with planner-generated expert demonstrations while accounting for efficiency and execution reliability.
- Evaluation Setup: Evaluation runs in AI2-THOR, selecting models with the lowest validation loss and terminating episodes after 1000 steps or more than 10 failed actions.Failed actions include wall collisions and incompatible object-mask predictions.
- Evaluation Metrics: ALFRED reports both Task Success and Goal-Condition Success, capturing complete task outcomes and partial completion of required conditions.Goal conditions can require multiple instructions, such as navigating to and slicing an object.
- Evaluation Metrics: Goal-Condition Success is the ratio of completed goal-conditions to required conditions, and Task Success is 1 only when that ratio equals 1.The average ALFRED task contains 2.55 goal conditions.
- Path Weighted Metrics: Path-weighted versions of both metrics adjust scores using expert demonstration length because planner-generated demonstrations are efficient but not guaranteed optimal.A model receives half-credit for taking twice as many actions as the expert.
6. Analysis
Analysis shows that the baseline performs poorly on ALFRED’s long-horizon, compositional tasks, although it completes some isolated sub-goals. Both visual and language inputs matter, while progress monitoring offers only modest gains and generalization remains weak.
- Overall Performance: ∼8% goal-condition success is achieved by the initial SEQ2SEQ model, which lacks maps, object segmentations, and explicit object-state tracking.The result indicates partial task completion but substantial headroom on ALFRED’s long-horizon tasks.
- Overall Performance: The baseline’s poor ALFRED performance contrasts with navigation datasets where sequence-to-sequence models with progress monitoring perform well.The analysis attributes the ALFRED difficulty to long-horizon tasks and high-dimensional state spaces.
- Baselines: 0% on all folds is obtained by an agent choosing uniformly random actions and interaction masks.This baseline is evaluated even without an API failure limit.
- Model Ablations: Both vision and language are necessary: unimodal ablations complete some goal-conditions but rely on memorized objects, navigation instructions, or interaction masks.The NO VISION model can memorize masks for common centered objects, while NO LANGUAGE can interact with familiar training objects.
- Language Ablations: Low-level instructions alone perform similarly to combined goal and step-by-step language, while goal-only language is insufficient for completing the tasks.The model does not appear to use goal language to plan sub-goals for step-by-step execution.
- Model Ablations: ∼1% to ∼2% success is gained from two progress-monitoring signals, which also produce consistently higher path-weighted scores.The signals may reduce action repetition and improve Stop-action prediction.
- Efficiency and Generalization: Models take more steps than experts in every case and generalize poorly to unseen scenes because of state-tracking failures and visual complexity.One example is repeatedly heating an egg in the microwave; unseen scenes also introduce novel object instances.
- Human Evaluation: 91% human task success and 86% path-length weighted success are obtained on 100 directives from the unseen test fold.Participants took slightly longer than experts, indicating alignment between directives and demonstrations.
7. Conclusions
ALFRED evaluates mapping natural language instructions and egocentric vision to action sequences, using environment dynamics and interaction masks to narrow the simulation-to-robot gap. A sequence-to-sequence baseline remains poor on overall tasks despite competence on some subgoals, exposing challenges in long-horizon compositional planning.
- ALFRED maps natural language instructions and egocentric vision to sequences of actions, including interaction-mask predictions for object manipulation.
- The benchmark narrows the gap between simulated agents and real-world robots through environment dynamics and pixel-level interaction requirements.
- Overall task success rates remain poor for the evaluated sequence-to-sequence model, although it is relatively competent on some isolated subgoals.The model faces visual semantic navigation, object detection, referring-expression grounding, and action-grounding challenges.
- ALFRED’s long task horizons create subproblems that may require hierarchical, modular, and structured reasoning or planning models.
Appendix A. Dataset Details
The dataset appendix describes ALFRED’s heterogeneous task sampling, expert-demonstration generation, and multi-level language annotation process. It documents task parameters, dataset distributions, annotation interfaces, and replayable demonstrations.
- Task and demonstration generation: ALFRED samples task type, scene, pickup object, destination, and secondary object parameters to construct heterogeneous household tasks.The secondary object applies to Stack & Place tasks, while the destination is absent for Examine.
- Task and demonstration generation: The sampling algorithm greedily targets imbalances across parameter slots, weakening priors such as ignoring language or always executing common tasks.
- Dataset statistics: The appendix reports train, validation-seen, and validation-unseen task distributions, subgoal distributions across seven task types, and vocabulary statistics.
- Task and demonstration generation: Expert episodes instantiate randomized scenes and object or agent positions, then use PDDL rules to generate demonstrations; invalid plans or failed actions are abandoned.
- Language annotation: After three annotations, additional annotators introduce fewer than 10 unique tokens on average, motivating the choice of three directives per demonstration.
- Language annotation: Annotators receive expert-demonstration videos with subgoal timeline segments and write segment-level directives plus a summary of the full sequence.
A.5. Dataset Examples
The dataset examples appendix presents seven expert trajectories, one for each task type, together with their accompanying language annotations.
- Seven expert trajectories illustrate ALFRED’s task types and their associated language annotations.
Appendix B. Implementation Details
The implementation appendix details the sequence-to-sequence baseline, its visual, language, action, and mask components, training procedure, replay support, and qualitative mask behavior. It also explains why random success is unlikely in ALFRED’s large action and state spaces.
- Training and data: The replayable generation pipeline preserves initialization information, enabling augmentation with high-resolution images, depth maps, or object-segmentation masks.
- Model architecture: A pretrained ResNet-18 produces 512 × 7 × 7 visual features, which a two-layer 1×1 CNN reduces to a 64 × 7 × 7 representation and then a 2500-dimensional vector.
- Model architecture: The language encoder is a bidirectional LSTM with hidden dimension 100, and self-attention initializes the decoder’s hidden state.
- Model architecture: The action decoder is an LSTM with hidden dimension 512, while the mask decoder outputs a 1×300×300 binary interaction mask through deconvolution and up-sampling.
- Evaluation behavior: ALFRED’s average branching factor is approximately 10^53 across 50 steps and 12 actions, making random achievement substantially unlikely.
- Evaluation behavior: In qualitative examples, microwave masks fit familiar seen-scene contours better than sink masks fit unfamiliar unseen-scene topology.
C.1. Performance by Task Type
Performance varies sharply by task type: the strongest model handles some short-horizon tasks but achieves almost no success on long-horizon compositions. Table A1 reports these success percentages across seven task types.
- The best model, SEQ2SEQ+PM, mostly succeeds on short-horizon Pick & Place and Examine tasks.
- Long-horizon Stack & Place and Pick Two & Place tasks have near-zero success rates across all models.
- Table A1 reports success percentages across seven task types, with the highest values highlighted in blue.