Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion

1 Introduction

• Task-completion agents often predict movement without explicitly identifying navigation targets; EmBERT grounds navigation instructions by predicting the goal object's location at each timestep.This adds object landmarks to navigation actions, alongside manipulation targets.Source: §1 · p. 1
• EmBERT decouples single-timestep language and visual embedding from transformer decoding over embedded-state sequences to predict the next action.The paper contrasts this with beam search over action sequences and recurrent memory approaches limited to short trajectories.Source: §1 · p. 1
• Object-centric navigation unifies navigation and interaction actions in ALFRED by assigning per-step object landmarks to navigation actions.The model predicts an action, target object, and target object's parent for navigation and manipulation.Source: §1 · pp. 1–2, Figure 1 · p. 2
• EmBERT's decoupled multimodal transformers extend sequence-to-sequence prediction to ALFRED trajectories averaging fifty steps.The introduction presents the model as addressing long temporal histories in embodied task completion.Source: §1 · p. 2

2 Related Work

• EmBERT adds navigation-object targets to ALFRED task completion, where prior task-completion models often predicted movement without explicitly identifying navigation targets.Navigation benchmarks commonly model objects as targets, and object modeling assists navigation success (Shrivastava et al., 2021; Qi et al., 2020a, 2021).Source: §2 · p. 2
• EmBERT decouples multimodal state embedding from transformer translation of state embeddings into action and object-target predictions.Table 1 contrasts approaches that maintain history through transformer states, subgoal structures, or voxel maps.Source: §2 · p. 2, Table 1 · p. 3
• EmBERT uses the BERT stack in its multimodal architecture rather than training its language encoder from scratch, and the authors identify VLN and Cooperative Vision-and-Dialog Navigation as possible applications.The paper also leaves mapping for future work, despite prior promise in VLN and ALFRED (Fang et al., 2019; Chen et al., 2021; Blukis et al., 2021).Source: §2 · pp. 2–3

3 The ALFRED Benchmark

• ALFRED tasks cover seven household categories and require object manipulation, placement on a receptacle, and sometimes intermediate state changes.Examples include heating an object in a microwave before placing it.Source: §3 · p. 3
• Each episode pairs room state and language instructions with planning goals and an expert action-and-object-mask trajectory aligned to instructions.An instruction can align with a navigation goal and a sequence of turning and movement actions.Source: §3 · p. 3
• EmBERT uses panoramic views and detected-object bounding boxes, selecting the box with the highest intersection-over-union with the demonstrated object mask.The additional views are collected by turning the agent after navigation actions.Source: §3 · p. 4
• For navigation actions, EmBERT labels the goal-class object in the instruction's final aligned frame as the target for every action in that instruction's trajectory segment.It also trains a parent-receptacle prediction so the agent can navigate to a visible landmark such as the table supporting a box.Source: §3 · p. 4

4 Embodied BERT

• EmBERT uses a transformer encoder to jointly embed language and visual tokens, followed by a transformer decoder for long-horizon planning and object-centric navigation predictions.The encoder combines goal and step instructions with object detections from panoramic views.Source: §4 · p. 4, §4.1 · p. 4, Figure 3 · p. 5

4.1 Multimodal encoder

• Object embeddings combine visual features, bounding-box coordinates and area, and object heading; the encoder fuses these with instruction tokens into temporally independent agent states.The model uses fixed detection counts of 36 for the front view and 18 for side views.Source: §4.1 · pp. 4–5

4.2 Segment-Level Recurrent Action Decoder

• The segment-level decoder caches prior segment representations as extended context, allowing self-attention over earlier timesteps while stopping gradients between consecutive segments.Trajectories are divided into temporal segments to balance long-range context with memory constraints.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3 Auxiliary tasks

• Auxiliary tasks train EmBERT to advance to the next instruction, predict navigation targets and their parent receptacles, and classify visible object regions.Visual-region prediction assumes objects associated with an instruction will be in view at the end of its sub-goal.Source: §4.3 · p. 6

Comparison to Other Models

• EmBERT is competitive on ALFRED leaderboard tests, surpasses all but ET and ABP on Seen performance, and beats only MOCA on Unseen generalization success.The authors report these comparisons at the time of writing.Source: §5 · pp. 7–8

Implementation Details

• Training uses 10-timestep segments and teacher forcing; the TransformerXL decoder has two layers, eight attention heads, and a 200-slot memory cache.The best epoch is selected using the validation Seen-set average of action and object-selection accuracy per timestep on gold trajectories.Source: §5 · p. 7

Action Recovery Module

• After a failed navigation action, recovery tries the next-most-confident navigation action; after a failed object interaction, it tries the next-highest-confidence box.An episode ends after 10 API action failures.Source: §5 · p. 7

Comparison to Other Models

• Evaluation distinguishes episode completion (Task success), the fraction of subgoals satisfied (GC success), and path cost relative to expert actions (path-weighted success).Seen and Unseen sets distinguish rooms seen or unseen during training.Source: §5 · p. 7
• EmBERT predicts object instances directly rather than using MOCA’s Instance Association in Time module, so it must learn temporal instance associations implicitly.The authors characterize MOCA’s module as an inference-time fix conditioned on the selected target’s object class.Source: §5 · p. 8

EmBERT Ablations

• Removing object-centric navigation prediction, limiting decoder memory to one prior timestep, using BERT rather than OSCAR initialization, or restricting vision to the front view lowers performance.The reported decreases occur across metrics for object-centric navigation and in both Seen and Unseen folds for the other changes.Source: §5 · p. 8, Table 3 · p. 8
• Parent-prediction and VRC losses do not improve standard EmBERT, but with nine side-view boxes they improve Unseen success and goal-condition metrics and Seen success rate.Similar performance with 18 and 9 side-view boxes across all three auxiliary losses suggests the larger configuration may be over-parameterized.Source: §5 · p. 8

6 Conclusions

• EmBERT adapts pretrained OSCAR and decouples transformer embedding from decoder steps, enabling attention to many features per timestep and a history of prior embedded states.The authors describe it as the first model to bring object-centric navigation to language-guided tasks involving manipulation and navigation.Source: §6 · p. 8
• Object-centric navigation and the ability to attend across a long time horizon both contribute to EmBERT’s competitive performance with state-of-the-art ALFRED models.The authors identify this relationship in their conclusion and refer to the ablation results in Table 3.Source: §6 · p. 8
• The authors propose applying EmBERT to other multimodal benchmarks and increasing ALFRED training data through randomized scenes or generated language supervision.The proposed future directions include vision-and-audio inputs, wider task arrays, and additional expert demonstrations generated with a speaker model.Source: §6 · pp. 8–9

• EmBERT is evaluated only on ALFRED, whose directives provide a one-sided “recipe” for accomplishing a task.The architecture may apply to single-instruction tasks such as VLN when auxiliary navigation object targets can be derived from data.Source: §7 · p. 9
• The architecture can apply to single-instruction tasks such as VLN when auxiliary navigation object targets can be derived from the data.For ALFRED, the paper derives those targets by treating step-by-step instructions as empty.Source: §7 · p. 9
• Simulation omits haptic feedback during manipulation and environmental audio and speech, while low-level physical robot control poses separate challenges from ALFRED’s abstract locomotion.Source: §7 · p. 9
• Because ALFRED evaluates written English, whether EmBERT’s object-centric navigation methods generalize to other languages remains uncertain.The authors also acknowledge possible pernicious priors and language prompt-based attacks in large-scale networks.Source: §7 · p. 9

• The alternative auxiliary-loss configurations did not produce results comparable with the best-performing model, motivating further analysis of loss design and combination.Source: §A.1 · p. 13

Masked Language Modeling

• Masked language modeling masks 15% of instruction tokens at a subgoal’s final step, assuming the associated objects are then in view.This timing addresses the possibility that instruction-mentioned objects are not visible at the trajectory’s first timestep.Source: §A.1 · p. 13

Masked Region Modeling

• Masked region modeling zeros 15% of visual features and asks the model to predict them from its time-dependent representations.Source: §A.1 · p. 13

Image-text Matching

• Image-text matching maximizes cosine similarity for current visual-language pairs and minimizes it against other batch instructions, using a contrastive loss.The authors expect instruction meaning to be appreciated through the agent-state representation after several timesteps of actions.Source: §A.1 · p. 13

• AI2-THOR uses the Apache-2.0 License; ALFRED uses the MIT License.Source: §A.2 · p. 13