Source-linked AI summary
FILM: Following Instructions in Language with Modular Methods
So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, Ruslan Salakhutdinov
TL;DR
Embodied instruction following requires agents to understand compositional instructions, navigate and interact over long horizons, and localize objects precisely, while end-to-end methods must learn several capabilities simultaneously. FILM addresses this with modular language processing, semantic mapping, semantic search, and deterministic action policies, achieving state-of-the-art performance on ALFRED without sequential supervision.
Problem
Embodied instruction following requires compositional instruction understanding, large action-space control over long horizons, and fine-grained object localization, while end-to-end methods must simultaneously learn state-tracking, spatial memory, exploration, planning, and control.
Method
FILM modularly processes language, converts egocentric vision into a semantic metric map, predicts a search goal location, and outputs navigation and interaction actions without requiring expert trajectories or low-level language instructions.
Results
24.46%: FILM achieves state-of-the-art performance on ALFRED, with an 8.17% absolute gain in SR on Tests Unseen over HLSM; adding low-level language increases success rate to 26.49%.
Takeaways & Limitations
Explicit structured spatial memory coupled with semantic search can provide better state-tracking and exploration without expert trajectories or low-level language instructions.
Takeaways & Limitations
FILM is not designed to search inside closed receptacles, although subgoal objects frequently appear in them; extending it requires more active search.
Abstract
from arXiv · showhide
Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 %) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 %). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.
1 INTRODUCTION
Embodied instruction following requires agents to understand compositional instructions, navigate and interact over long horizons, and localize objects precisely. FILM addresses the limitations of end-to-end methods with modular structured representations and achieves state-of-the-art performance without sequential guidance.
- Motivation: EIF combines language understanding, navigation, interaction, long-horizon execution, and fine-grained object localization.Its challenges extend beyond navigation to executing sequences of subtasks from language instructions.
- Motivation: End-to-end methods must jointly learn state tracking, spatial memory, exploration, long-term planning, and low-level control.Existing approaches commonly rely on neural memory and expert trajectories over language and visual inputs.
- FILM: FILM decomposes instruction following into language processing, semantic mapping, semantic search, and deterministic navigation or interaction.The design uses structured spatial components and does not require expert trajectories or low-level language instructions.
- Results: 24.46% state-of-the-art performance on ALFRED exceeds the previous SOTA by 8% absolute, while low-level language raises success rate to 26.49%.The method achieves this result without expert trajectories or low-level instructions by default.
2 RELATED WORK
Related EIF methods range from end-to-end sequence models to modular systems with structured language or spatial representations. FILM differs by making spatial guidance explicit and requiring neither low-level instructions nor expert trajectories.
- Prior embodied tasks: VLN methods address navigation to a goal area, whereas EIF additionally requires sequences of navigation and interaction subtasks.EIF therefore combines broader task execution requirements than standard vision-language navigation.
- End-to-end EIF: End-to-end EIF baselines concatenate language, visual features, and past trajectories to predict subsequent actions.Seq2Seq and episodic-transformer approaches retain action prediction as an end-to-end process.
- Modular EIF: Other EIF systems modularize language and visual processing but retain neural action modules that implicitly track spatial, progressive, and agent states.FILM instead uses structured language and spatial representations that make failure reasons more transparent.
- Structured spatial memory: Blukis et al. introduced structured spatial memory but sampled navigation goals from a model trained on expert interaction-pose labels.FILM adds explicit semantic search and removes dependence on expert trajectories and low-level instructions.
- Semantic mapping: Semantic mapping has also been used in navigation, but the cited object-goal-navigation setting is simpler than EIF.These methods build semantic maps using differentiable projection operations.
3 TASK EXPLANATION
ALFRED evaluates agents that execute household tasks from natural-language instructions and egocentric vision. Tasks vary in complexity and require completing all subtasks within fixed action and step limits.
- Benchmark: ALFRED agents receive natural-language instructions and egocentric vision while completing household tasks.An example requires picking up an apple, heating it in a microwave, and placing it on a countertop.
- Task structure: The benchmark includes seven task types ranging from Pick & Place to more complex Heat & Place tasks.Instructions can also require operations such as slicing an object before subsequent actions.
- Evaluation: An episode is successful only when all subtasks are completed within 10 failed low-level actions and 1000 maximum steps.
4 METHODS
FILM combines learned language processing, semantic mapping, and semantic search with a deterministic policy for navigation and interaction. Its map represents observed objects and spatial state, while search predicts likely locations for unseen subgoal objects.
- Architecture: FILM uses three learned modules—Language Processing, Semantic Mapping, and Semantic Search Policy—plus a deterministic navigation and interaction policy.Language is processed initially, mapping updates every time step, and search goals are produced at a coarse time scale.
- Language Processing: The Language Processing module predicts instruction type and arguments, then fills a matching template with the predicted arguments.Arguments include objects, receptacles, slicing status, and parent objects for intermediate movable receptacles.
- Semantic Mapping: Semantic Mapping converts egocentric RGB into depth and instance segmentation, projects the results into a point cloud, and aggregates them into a voxel-based semantic map.The map is locally updated and accumulated over time.
- Semantic Mapping: The allocentric semantic map is a (C + 2) × M × M binary grid whose cells represent 5cm × 5cm scene regions.Object-category channels are supplemented by obstacle and exploration channels.
- Semantic Search: The Semantic Search Policy predicts a coarse 2D distribution over likely locations of a small subgoal object using a map containing receptacle objects.This addresses tiny objects, limited downward field of view, and segmentation failures at some viewing angles.
- Semantic Search: The search policy is trained without expert trajectories by collecting receptacle-only maps and subgoal locations every 25 steps.A CNN predicts a coarse grid distribution, trained by minimizing KL divergence to a pseudo-ground-truth distribution.
- Semantic Search: Predicted search distributions allocate probability across plausible furniture for bowls and concentrate near likely faucet locations such as sinks and bathtub ends.The examples illustrate uncertainty over multiple plausible receptacles rather than a single point estimate.
- Deterministic Policy: The deterministic policy selects the observed target when available; otherwise, it navigates toward a sampled search goal before performing the required interaction.Navigation uses the Fast Marching Method, and successful interactions advance the subtask pointer.
5 EXPERIMENTS AND RESULTS
FILM is evaluated on ALFRED using success, goal-condition, and path-length-weighted metrics across seen and unseen scenes, with ablations and error analyses examining its modules and semantic search policy. Results show strong performance and generalization, while perception, object localization, and task-specific scene factors remain important limitations.
- Evaluation: SR is a binary indicator of completing all subtasks, while GC measures the ratio of goal-conditions completed at episode end.The test set separates Tests Seen from Tests Unseen, and the official leaderboard ranks entries by SR on Tests Unseen.
- Main results: 8.17% absolute (50.15% relative) gain in SR on Tests Unseen and 0.66% absolute (2.63% relative) gain on Tests Seen over HLSM establish FILM’s test-time advantage.FILM achieves state-of-the-art performance across both seen and unseen scenes when only high-level instructions are provided.
- Main results: 11.06% absolute (71.68% relative) gain in SR on Tests Unseen over ABP is achieved when FILM also receives low-level instructions.FILM performs competitively against methods requiring step-by-step instructions and performs similarly across Tests Seen and Tests Unseen.
- Semantic search: Semantic search significantly increases SR, GC, and their path-length-weighted versions, with the larger PLWSR gap suggesting more efficient trajectories.The improvement from the semantic policy is reproduced across multiple random seeds.
- Ablations and errors: 9.26% / 9.26% gains from ground-truth segmentation identify segmentation as a bottleneck in seen and unseen scenes, while 10.64% depth improvement is specific to unseen scenes.Ground-truth language produces SR gains of 7.43% / 4.22%, whereas adding low-level language yields a comparatively incremental improvement.
- Ablations and errors: Goal object not found is the most common error mode, often because objects are small or viewpoint-dependent; closed receptacles, interaction, navigation, and language processing also cause failures.The semantic search policy alleviates this error in certain cases, especially for small and flat objects in large scenes.
6 CONCLUSION
FILM is a modular method for embodied instruction following that achieves state-of-the-art performance on ALFRED without sequential supervision. The research targets autonomous agents, but applications in human environments should consider privacy, and models trained on Ai2Thor may reflect North American homes.
- FILM achieves state-of-the-art performance on the ALFRED benchmark without any sequential supervision.
- FILM processes language into structured forms, builds a semantic metric map, predicts likely goal locations, and outputs navigation or interaction actions.
- The research is intended for building autonomous agents.
- Applications in environments with humans should take privacy concerns into consideration.
- Models trained using Ai2Thor may be biased towards North American homes.
A.1 TASK DEFINITION
ALFRED requires agents to complete household tasks from high- and low-level language instructions, combining navigation with interaction sequences. The benchmark includes seven templated task types.
- The benchmark contains seven task types with subtask sequences templated according to task type.
- ALFRED provides agents with high-level and low-level language instructions for achieving household goals.
- Success requires completing a sequence of interactions and the navigation between those interactions.
A.2 SEMANTIC MAPPING MODULE
The semantic mapping module transforms egocentric visual observations into semantic voxels and aggregates them into a global map. Object categories are selected dynamically for efficiency.
- Depth and instance segmentation predictions from egocentric RGB are transformed into semantic voxels and summed across height into a semantic map.
- Partial maps from different time steps are aggregated into a global map using sum or logical-or operations.
- The number of mapped object categories is dynamically controlled because the environment contains more than 100 objects.
A.3 SEMANTIC SEARCH POLICY MODULE
The semantic search policy predicts search locations over an 8 × 8 grid for the object currently being sought. At deployment, the agent samples from that object's output channel.
- The semantic search policy processes the map through convolutional layers and predicts an 8 × 8 grid over 73 object-category channels.
- During deployment or validation, the agent samples a search location from the channel corresponding to the object currently being searched for.
- The semantic search policy is presented as a dedicated module for guiding exploration.
A.4 IMPACT OF GRID SIZE ON THE EFFECTIVENESS OF THE SEMANTIC SEARCH POLICY
The semantic search policy’s grid size trades off label sparsity against spatial precision. The preferred coarse-cell size should depend on scene/task characteristics and the frequency of goal objects.
- Grid-size trade-off: A finer grid is harder to train because labels become sparse, whereas a coarser grid spreads probability mass across widely separated locations.The grid therefore balances trainability and localization precision.
- Goal frequency: More frequent goal objects make the discrepancy between actual and coarse ground-truth distributions more tolerable.The bound is more favorable when the goal object occurs frequently.
- Goal frequency: Infrequent goal objects can make the actual and coarse distributions too different in the worst case, so L and N should reflect average goal-object frequency.Adaptive grid sizing is identified as future work.
A.5 PSEUDOCODE FOR THE DETERMINISTIC POLICY
The appendix specifies FILM’s deterministic navigation and interaction procedure and documents supporting implementation and evaluation details. It also examines reproducibility and the consequences of removing the template assumption from language processing.
- Deterministic policy: The deterministic policy selects the nearest observed subgoal object, or a semantic-search goal when the object is not currently mapped, then navigates with FMM.After reaching the stopping distance, the policy continues with rotation and interaction decisions.
- Implementation details: The implementation defines visibility using the RGB frame and a predicted-depth threshold of 1.5 meters, while FMM denotes Fast Marching Method.MoveBehind, SideStep, and RotateBack are composite actions defined by the authors rather than native ALFRED actions.
- Deterministic policy: Algorithm 1 takes a list of goal tuples and iteratively updates the semantic map while executing navigation and interaction until task success or failure.The episode loop is bounded by 1,000 timesteps.
- Reproducibility: The semantic-policy improvement is reproducible across multiple starting seeds, with Table 8 reporting Tests Unseen results and sample variance for the average.The policy was trained with learning rate 0.001 and evaluated every 50 steps under a stated model-selection rule.
- Language processing: Replacing templates with a BART mapping from high-level instructions to subtask sequences reduced valid-unseen success rate from 20.10% to 18.03%.The comparison uses ALFRED subtask-sequence annotations rather than the original template assumption.
- Language processing: Templates were created from seven training episodes in under 20 minutes and covered 20,000 training episodes, while general instruction following still requires better program synthesis beyond fixed instruction types.The authors therefore view templates as cheaper than subtask annotations but not sufficient for unrestricted instruction types.