Skill Induction and Planning with Latent Language

1 Introduction

• Hierarchical learning remains difficult because formal task specifications limit applicability, while simple high-level actions limit expressive power.The broader challenge is integrating high-level reasoning with low-level perception and control.Source: §1 · p. 1
• Earlier language-based policy learning was limited to short horizons, restricted language, or synthetic data, with training requiring large fully supervised alignment datasets.Existing approaches treat language as latent only at prediction time.Source: §1 · p. 2
• A latent-plan model lets goals generate subtask descriptions and descriptions generate actions, trained through inference over combinatorial plans seeded by a small annotated set.Training uses ungrounded text, unlabeled demonstrations, and annotations linking them.Source: §1 · p. 2
• With 10% of labeled training instructions, the approach matches a state-of-the-art model with stronger dataset-specific assumptions and outperforms methods using more training and evaluation information.The comparison is on a reduced ALFRED dataset, with no training alignments and no evaluation instructions.Source: §1 · p. 2
• The results indicate that a rich language model, many demonstrations, and a small number of annotations suffice for compositional skill libraries and effective policies.The approach also automatically segments and labels unlabeled demonstrations, including subtasks composed from novel actions and objects.Source: §1 · p. 2

2 Preliminaries

• The learning objective is a goal-conditional policy that generalizes demonstrated behavior to novel goals and states in episodic, partially observed environments.Demonstrations pair goals with observation-action sequences; the transition and observation functions are unknown.Source: §2 · p. 2
• A hierarchical policy first generates subtask specifications from the goal, then generates actions for each specification until STOP returns control to the controller.The executor conditions on action and observation history and the current subtask.Source: §2 · p. 2
• Prior hierarchical approaches either predefine and independently supervise a skill inventory or discover a finite inventory through clustering.The section motivates richer supervision to guide skill discovery beyond these approaches.Source: §2 · p. 3
• Language offers a structured, expressive space for planning, but current latent-language policy training requires large datasets with independent fine-grained supervision for controller and executor.The paper instead proposes learning from unlabeled demonstrations with a small amount of natural-language supervision.Source: §2 · p. 3

• The algorithm jointly partitions unannotated action sequences into reusable segments, labels them with inferred language descriptions, and trains goal-to-subtask and subtask-to-action policies.Source: §3 · p. 3
• Training combines unannotated demonstration-goal pairs with fewer annotated triples; annotated trajectories omit segment boundaries and STOP markers.Source: §3 · p. 3
• Maximum likelihood jointly imputes missing segmentations and labels while fitting controller and executor parameters, alternating re-segmentation, relabeling, and parameter updates.Source: §3 · p. 4
• Alignment search uses the executor's dependence only on the current subtask to compute highest-scoring segmentations in O(|d||τ|) space and O(|d|^2|τ|) time.Source: §3 · p. 4
• Because meaningful boundaries require a trained executor, the first iteration initializes segment boundaries from transitions in a 3-state HMM.Source: §3 · p. 4
• A learned amortized proposal model infers each unlabeled segment's description from its actions, the following segment's actions, and the goal; annotated labels stay fixed.Source: §3 · p. 5

• ALFRED provides 8,055 demonstrations across 120 rooms and 25,743 English instructions, with image observations and 12 navigation and manipulation primitives.Source: §4 · p. 6
• ALFRED evaluation gives agents goals but not fine-grained instructions at test time, focusing on learning from partial language annotations without segmentations or alignments.Source: §4 · p. 6
• The policies use T5-small-initialized sequence-to-sequence transformers for goal-to-instruction planning and observation-conditioned instruction-to-action execution.Source: §4 · p. 6
• For goal-only navigation, detailed directions are replaced with generic labels such as 'find a glass' because training demonstrations do not show exploration.Source: §4 · p. 6
• Modular variants execute Find subtasks with either a planner using ground-truth environment information or the HLSM navigation module.Source: §4 · p. 6
• Baselines include a non-hierarchical goal-conditioned policy, a supervised hierarchical policy, and ablations of pretraining and latent-label learning.Source: §4 · p. 6
• The study also compares against ALFRED instruction-following models, many trained with instructions and alignments or ground-truth test instructions.Source: §4 · pp. 6–7
• Online evaluation measures subtask accuracy and end-to-end success; data-efficiency experiments use offline exact-match accuracy on predicted subtask action sequences.Source: §4 · p. 7

Main results

• (SL)3 improves subtask completion by 25% over seq2seq in both annotation conditions and achieves end-to-end performance comparable to HLSM while outperforming several more-supervised methods.The comparison includes methods with ground-truth training segmentations and alignments or ground-truth test-time plans.Source: §5 · p. 7, Table 1 · p. 7

Sparse supervision

• Performance with 10% of natural-language annotations is similar to performance with 100%, and both conditions outperform fully annotated seq2seq and seq2seq2seq baselines.The reported comparison concerns policy performance; the baselines use 100% of natural-language annotations.Source: §5 · p. 7

Ablations

• Without latent-instruction inference, performance drops from 56% to 52%, while an inferred example segments an unannotated trajectory and labels its steps.Figure 3 illustrates a 10-segment inferred trajectory with labels consistent with objects, receptacles, and subtasks.Source: §5 · p. 7, Figure 3 · p. 8
• At 10% annotation, pretrained models complete 56% of tasks versus 49% from scratch; Table 2 attributes 7% and 4% of completion rate to pretraining and latent-task inference, respectively.Ground-truth training alignments also improve task completion rates.Source: §5 · p. 7, Table 2 · p. 8
• The experiments point to a potentially broad role for pretrained language models in tasks without language input or output.The paper attributes some benefit to pretrained models encoding common-sense plan structure.Source: §5 · pp. 7–8

Generalization

• The model produces correct or plausible high-level plans for many out-of-distribution goals, though generating their low-level actions needs further modeling.The reported goals include novel objects or actions outside the ALFRED dataset.Source: §5 · p. 8

Language-based supervision and representation

• Unlike policy-sketch approaches with fixed skill inventories, (SL)3 learns an open-ended, compositional library of behaviors indexed by natural-language strings.The paper uses language annotations to discover compositional subtask abstractions for autonomous control.Source: §6 · p. 8

Hierarchical policies

• Hierarchical policy learning has used direct supervision or manual skill specifications, as well as unsupervised discovery; this approach seeks fine-grained skill control with less human supervision.Prior examples cited include Sutton et al. (1999), Kulkarni et al. (2016), Dietterich (1999), and Bacon et al. (2017).Source: §6 · p. 8

Language and interaction

• Language-and-control work spans instruction following, embodied question answering, and dialogue, and the authors suggest extending semi-supervised plan-description inference to these settings.The cited examples include Chen and Mooney (2011), Das et al. (2018), and Tellex et al. (2020).Source: §6 · p. 9

Conclusion

• (SL)3 infers the latent structure of unannotated demonstrations, segmenting them into subtasks and labeling each with a compositional description.Its policy uses language as an abstract representation of subgoals and plans: a controller maps goals to plan specifications, and a modular executor maps plan components to low-level actions.Source: §7 · p. 9
• In simulated household environments, the model completes abstract goals with accuracy comparable to state-of-the-art models trained and evaluated with fine-grained plans.The cited example contrasts an abstract goal such as “slice a tomato” with detailed plans such as finding and carrying a knife.Source: §7 · p. 9
• The authors identify image understanding, program synthesis, and language generation as domains with hierarchical structure where generalized versions might use limited rich language supervision.The proposed extension is framed as a possibility, not an evaluated result in those domains.Source: §7 · p. 9

A Out-of-distribution Generalization

• The evaluation probes open-endedness through novel object names, verbs, skill combinations, and out-of-distribution prompts.The paper notes that (SL)3 does not require a prespecified fixed inventory of goals or actions.Source: A Out-of-distribution Generalization · p. 12
• The model correctly orders heating and slicing for an unseen “slice a heated apple” goal and generalizes to combinations such as cleaning and cooling an apple.In training, objects were only heated after being sliced.Source: A Out-of-distribution Generalization · p. 12
• It shows some success with novel objects such as carrots and masks, and with related verbs such as scrub, but fails on unrelated verbs such as squash.For the carrot example, the first subtask incorrectly refers to lettuce, while later subtasks refer to carrot.Source: A Out-of-distribution Generalization · p. 12
• Affordances and constraints are incompletely modeled, and plans with considerably more subgoals than training goals may skip subtasks.For example, the model plans to slice a bowl in response to an impossible goal and may omit skills in very long sequences.Source: A Out-of-distribution Generalization · p. 12

B Initialization: Segmentation Step

• A randomly initialized πE yields extremely low-quality segmentations, so the authors instead initialize from unsupervised low-level action segmentation.The supplied passage says πE cannot be initialized by training on Dann.Source: B Initialization: Segmentation Step · p. 12
• The initialization uses Baum–Welch to fit a hidden Markov model to low-level action sequences, with three hidden states in the reported experiments.Transition and emission distributions and maximum a posteriori sequence labels are obtained using expectation–maximization.Source: B Initialization: Segmentation Step · p. 12
• 87.9% action-level accuracy was achieved against ground-truth segmentations using the initial unsupervised segmentations.The initial segmentations were obtained by inserting boundaries at transitions between distinct hidden states.Source: B Initialization: Segmentation Step · p. 12

C Model Architecture: Details

• The controller policy πC is a fine-tuned T5-small model.The executor policy πE decodes low-level action sequences conditioned on the agent’s first-person visual observations.Source: C Model Architecture: Details · p. 12
• Low-level actions are converted into templated natural-language commands and parsed to select actions for the ALFRED simulator.For example, put(cup,table) becomes “put the cup on the table.”Source: C Model Architecture: Details · p. 12
• For manipulation, a MaskRCNN model supplies action masks, and the mask labeled with the generated object name is selected.Both policies are optimized with AdamW using a learning rate of 1e-4, weight decay of 0.01, and ϵ = 1e-8.Source: C Model Architecture: Details · p. 12

D Role of trajectory length

• The executor generates actions token by token, conditioning each prediction on visual input and the preceding token; manipulation uses object masks, while navigation does not.After each action is executed, the environment returns an updated visual state, and generation continues until STOP.Source: Figure 6 · p. 13
• 55.6% success rate for (SL)3, 56.7% for seq2seq2seq, and 53.6% for seq2seq followed navigation replacement with TeleportTo.The modification reduced average trajectory length from 50 actions to 9.Source: D Role of trajectory length · p. 13
• In the shortened-plan setting, hierarchical policies remain useful but by a smaller margin; the authors suggest modeling navigation skills and their conditional independence is crucial for long-horizon learning.The models and baselines perform quite poorly at navigation skills.Source: D Role of trajectory length · p. 13

E Navigation Instructions

• ALFRED’s post-hoc navigation directions depend on environment layout and cannot serve high-level planning without advance layout knowledge.The passage says successful behavior in novel environments requires exploration or explicit access to the environment’s layout.Source: E Navigation Instructions · p. 13
• Navigation directions are replaced with “Go to the [object]” or “Find the [object]” templates, using target object names from PDDL plans.Figure 7 shows examples of the modified annotations.Source: E Navigation Instructions · p. 13, Figure 7 · p. 13
• Table 3 compares the information available to models and baselines at training time and inference.Source: Table 3 · p. 14