PaperlayerView exact revision on arXiv

Source-linked AI summary

EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL

Thomas Carta, Pierre-Yves Oudeyer, Olivier Sigaud, Sylvain Lamprier

arXiv:2206.09674v4cs.CLcs.AIcs.LG

TL;DR

Sparse rewards make long-horizon language-conditioned RL sample-inefficient, and reward shaping often relies on expert-designed objectives. EAGER generates goal-derived questions, evaluates answers from trajectories, and provides confidence-weighted intrinsic rewards; its BabyAI experiments report robustness and sample-efficiency benefits without task-specific expert knowledge.

  • Problem

    Language-conditioned RL can be sample-inefficient with sparse rewards, and existing reward-shaping methods require expert input to define objectives, completion, or rewards.

    Source: §1 · p. 1

  • Method

    EAGER masks words in the language goal to generate auxiliary questions, then rewards correct answers inferred from the agent’s trajectory in proportion to confidence.

    Source: §1 · p. 2

  • Results

    EAGER retains ELLA’s good properties without task-specific expert knowledge, with BabyAI experiments reporting robustness and sample efficiency against ELLA and RIDE.

    Source: §1 · p. 3

  • Takeaways & Limitations

    EAGER provides an automated way to generate auxiliary objectives and intrinsic rewards without expert intervention.

    Source: §1 · p. 3

    EAGER assumes access to successful demonstrated behaviours paired with language commands to pre-train its question-answering module.

    Source: §3 · p. 4

Abstract

from arXiv · show

Reinforcement learning (RL) in long horizon and sparse reward tasks is notoriously difficult and requires a lot of training steps. A standard solution to speed up the process is to leverage additional reward signals, shaping it to better guide the learning process. In the context of language-conditioned RL, the abstraction and generalisation properties of the language input provide opportunities for more efficient ways of shaping the reward. In this paper, we leverage this idea and propose an automated reward shaping method where the agent extracts auxiliary objectives from the general language goal. These auxiliary objectives use a question generation (QG) and question answering (QA) system: they consist of questions leading the agent to try to reconstruct partial information about the global goal using its own trajectory. When it succeeds, it receives an intrinsic reward proportional to its confidence in its answer. This incentivizes the agent to generate trajectories which unambiguously explain various aspects of the general language goal. Our experimental study shows that this approach, which does not require engineer intervention to design the auxiliary objectives, improves sample efficiency by effectively directing exploration.

1 Introduction

EAGER uses question generation and answering to derive auxiliary objectives and intrinsic rewards from language goals, addressing sparse-reward learning and expert-dependent reward shaping. Its BabyAI experiments report sample-efficiency and robustness benefits without task-specific expert knowledge.

1 Introduction

  • Earlier methods either strictly decomposed goals into ordered sub-goals or used ELLA’s more flexible auxiliary objectives, but still needed expert input.Strict decomposition assumes a goal can be reached through an exact series of low-level policies, which may fail when primitive actions are unknown.Source: §1 · p. 2
  • Language-conditioned RL can be sample-inefficient when rewards are sparse, while existing shaping methods require experts to engineer objectives, completion criteria, or rewards.The engineering burden becomes demanding in more complex environments.Source: §1 · pp. 1–2
  • EAGER adapts reference-less QG/QA metrics: it masks words in the language goal, asks the agent to recover them from its trajectory, and rewards correct answers by confidence.The QA module is trained on successful example trajectories; the summed intrinsic rewards measure trajectory quality relative to the goal.Source: §1 · pp. 2–3
  • The paper presents a QG/QA trajectory-quality metric and an automated framework for generating auxiliary objectives and intrinsic rewards without expert intervention.The method assumes access to demonstrated behaviours paired with language commands to pre-train QA.Source: §1 · p. 3
  • EAGER retains ELLA’s good properties without task-specific expert knowledge, and BabyAI experiments report robustness and sample efficiency against ELLA and RIDE.The experiments use example trajectories to train QA, which the paper describes as more parsimonious than training an agent with behavioural cloning.Source: §1 · p. 3

2 Related work

Related work uses language for instruction-following, hierarchical abstraction, reward shaping, intrinsic exploration, and question-asking. EAGER draws on flexible auxiliary objectives and QG/QA evaluation while automating objective generation.

Language-conditioned RL

  • Language-conditioned RL uses instructions to formulate tasks, and BabyAI supports this setting with synthetic combinatorial language and procedural generation.The platform is used to decouple exploration challenges from perception challenges.Source: §2 · p. 3

Language as an abstraction in hierarchical RL

  • Hierarchical methods use language to define low-level goals or decompose high-level tasks, while ELLA relaxes strict decomposition with auxiliary objectives that EAGER automates.EAGER adopts this flexible framework but automates the decomposition.Source: §2 · p. 3

Language for exploration and reward shaping

  • Language-based reward-shaping approaches map instructions or narrations to intermediate rewards, while ELLA makes fewer assumptions about environment and task structure.Reward shaping supplies additional rewards to guide learning.Source: §2 · p. 3
  • Other exploration methods use language descriptions of states for novelty rewards or learn language-parameterized rewards through interaction; EAGER does not require state descriptions.The cited approaches include oracle-provided state descriptions and socially learned reward functions.Source: §2 · pp. 3–4

Asking questions in RL

  • Question-asking methods use questions to identify subtasks, prune actions, or obtain world knowledge, while NLP QG/QA metrics assess generated text without human references.EAGER draws on QG/QA methods that measure how much source information a generated text preserves.Source: §2 · p. 4

3 Problem statement

The problem statement frames language-conditioned RL as sparse-reward learning under a limited horizon and motivates automated shaping. It formalizes auxiliary objectives and their trajectory-based completion estimates using successful demonstrations.

3 Problem statement

  • The setting is language-conditioned RL with sparse rewards and at most H steps, conditions that make exploration difficult.The goal-conditioned reward is 1 within a specified distance threshold and 0 otherwise.Source: §3 · p. 4
  • Reward densification uses auxiliary objectives, but prior selection of relevant objectives required expert annotations and example trajectories, motivating automation.Experts also had to determine when objectives were achieved and assign appropriate rewards.Source: §3 · p. 4
  • The formal setup assumes successful trajectories paired with instructions, a function f that proposes auxiliary goals, and a function h that estimates their achievement probability from trajectories.The paper trains h on demonstrations so it can generalize to unseen trajectories.Source: §3 · p. 4
  • Behavioural cloning trains an agent to copy an expert from example trajectories, whereas EAGER is described as more parsimonious and able to reward trajectories beyond demonstrated areas.The paper attributes this distinction to EAGER’s generalization and points to Appendix C.Source: §3 · p. 5

4 Method

EAGER turns language goals into automatically generated auxiliary objectives and rewards trajectories that let a QA system recover goal information. Its confidence-weighted intrinsic rewards guide exploration while a policy-invariant transformation preserves successful-trajectory returns.

4.1 EAGER

  • The QG module automatically creates auxiliary objectives by masking words in the language goal, such as nouns and adjectives.The questions are interpretable and do not require task- or environment-specific expert knowledge.Source: §4.1 · p. 5
  • The QA module uses trajectory observations and actions to answer generated questions, with a “no_answer” option when the trajectory lacks enough information.It is pretrained on successful example trajectories; answered questions are removed from the active set to avoid repeated rewards.Source: §4.1 · pp. 5–6
  • Intrinsic reward scales with the QA system’s confidence, favoring clear trajectories that make goal information easier to recover.The framework adapts question-generation and question-answering ideas to compare trajectories against a language goal.Source: §4.1 · pp. 5–6
  • The reward transformation subtracts shaping reward at the final step of a successful trajectory so its return matches the unshaped return.The cited ELLA result establishes policy invariance; unsuccessful trajectories can still receive shaping guidance.Source: §4.1 · p. 6
  • The algorithm generates questions at episode start and gives a confidence-weighted bonus when the QA module answers an active question correctly.The scaling factor λ is tuned so unsuccessful trajectories cannot receive more reward than a successful trajectory from the optimal policy.Source: §4.1 · p. 6, N end function · p. 7

4.2 A particular instance of the method in the BabyAI framework

  • In BabyAI, EAGER uses expressive instructions to generate questions by masking nouns and adjectives, while the recurrent agent processes observation sequences.BabyAI tasks give reward only for successful completion within a limited number of steps.Source: §4.2 · p. 7

5 Experiments

Across BabyAI tasks, EAGER performs better than selected baselines and shows sample-efficiency benefits, including on a temporally constrained sequence task. Experiments also examine QA training distributions and design choices that affect reward shaping.

5.1 Experimental settings

  • Experiments use BabyAI tasks of varying complexity and compare EAGER with ELLA, RIDE, and PPO without reward shaping; EAGER requires less expert intervention than ELLA.The QA is trained on successful bot-generated trajectories, including noisy examples, and the policy experiments use PPO with four random seeds.Source: §5.1 · pp. 7–9

5.2 How does EAGER perform when sparsity increases?

  • EAGER outperforms ELLA on PutNextTo and Unlock, and performs significantly better than RIDE on PutNext and Sequence and slightly better on Unlock.On Unlock-Medium, it overcomes the key-pickup bottleneck without expert knowledge and achieves similar or better sample efficiency on most tasks.Source: §5.2 · p. 9

5.3 How does EAGER perform with a sequence of tasks under a temporal constraint?

  • On Sequence, EAGER performs better than RIDE and ELLA despite the task’s temporal constraint, which neither method’s auxiliary-objective decomposition preserves.EAGER progresses slowly at first when noisy trajectories lead the QA module to overuse “no_answer.”Source: §5.3 · p. 9

5.4 Is EAGER robust to QA performance?

  • QA success rate above 0.56 is associated with faster agent learning, while wide-distribution QA training learns faster than narrow-distribution training.For QA with success rate at or below 0.56, success rate first rises and later declines toward its pretrained level as the agent learns to complete the task.Source: §5.4 · p. 10

5.5 How do design choices on the QA module affect EAGER’s performance?

  • Both adding a “no_answer” response and using confidence-weighted rather than binary intrinsic rewards independently improve sample efficiency.The ablations compare EAGER with variants that remove either or both design choices.Source: §5.5 · p. 10

6 Conclusion

EAGER uses language abstraction to shape rewards in long-horizon, sparse-reward tasks by generating questions from goals and rewarding correct answers. The method assumes pretraining on example trajectories and was tested in BabyAI, a 2D environment with synthetic language.

6 Conclusion

  • EAGER generates questions from the goal and rewards the agent for answering them correctly, creating a curriculum over auxiliary objectives.The conclusion contrasts this with ELLA, which requires expert knowledge to choose auxiliary objectives, and notes that EAGER does not use an oracle for linguistic state descriptions.Source: §6 · p. 10
  • The method assumes a preexisting set of example trajectories to pre-train its QA system.Future work includes investigating online QG/QA strategies to remove this limitation.Source: §6 · p. 10
  • EAGER was tested on BabyAI, a 2D environment with synthetic language, with future work proposed for more complex language and realistic environments.The authors identify ALFRED and human instructions as a possible direction.Source: §6 · p. 10

A Tasks description and assumptions used for the different method of reward shaping

Table 2 describes the experimental tasks and indicates whether each is used to train the QA module or the agent.

A Tasks description and assumptions used for the different method of reward shaping

B Training details

The section covers QA training-data creation, the RL agent architecture, and baseline details.

B Training details

B.1 QA architecture and pre-training

The QA module uses question, observation, and action information from trajectories, with full trajectories for training and partial trajectories at test time. Its training data and sampling procedures are designed to support varied trajectories and include examples for unanswerable questions.

B.1 QA architecture and pre-training

  • The QA is trained on 7,500 procedurally generated example trajectories for each of four tasks, including a composed Sequence task.The mixed tasks are intended to encourage language compositionality and let QA ground a sequential instruction across PutNextTo and Open.Source: §B.1 · p. 14
  • At test time, the QA processes the active questions using the trajectory so far, while training uses full trajectories.The architecture combines a language encoder, a CNN for observations, and an action lookup table; the illustrated example is at the fifth episode step.Source: §B.1 · p. 14, Figure 6 · p. 15
  • Unanswerable questions are generated by pairing trajectories with questions from unrelated goals, retaining examples probabilistically according to goal-word overlap.The keep probability is chosen to favor negative examples when the goals share many words; its hyperparameters target a number of no_answer questions similar to ordinary questions.Source: §B.1 · pp. 14–15
  • Training trajectories mix unperturbed bot behavior with randomly substituted actions at probabilities p = 0.1, 0.4, or 0.8, discarding trajectories the bot cannot complete.The distribution includes p = 0 with probability 0.45 and the three noisy settings with probabilities 0.35, 0.1, and 0.1, respectively.Source: §B.1 · p. 15
  • The QA model is trained with cross-entropy, batch size 10, Adam, and an initial learning rate of 10^-4 reduced by a factor of 0.1 every five epochs.Source: §B.1 · p. 15

B.2 Agent architecture and training

The agent uses BabyAI’s actor-critic PPO setup, while EAGER and the comparison methods add different forms of reward shaping. The appendix describes RIDE’s intrinsic-reward implementation and ELLA’s learned auxiliary objectives, which require expert-labelled trajectories and objective selection.

B.2 Agent architecture and training

  • The BabyAI agent uses an actor-critic architecture with PPO; its actor outputs action probabilities and its critic estimates the current state’s value.The six actions are turn left, turn right, forward, pick up, drop, toggle, and done.Source: §B.2 · p. 15
  • The action-generation procedure allows a random action with probability p at each time step.Source: Figure 7 · p. 16
  • The actor-critic network combines language and image features with FiLM, processes observation history with an LSTM, and feeds actor and critic heads.The figure attributes FiLM to Perez et al. [2017].Source: Figure 8 · p. 16

B.3 RIDE

  • RIDE rewards actions that significantly change the environment in a learned latent space; the authors adapt it to PPO’s on-policy rollouts.They use published dynamics-loss coefficients 10 and 0.1, learning rate 10^-4, and intrinsic-reward coefficient λ = 0.5.Source: §B.3 · p. 16

B.4 ELLA

  • ELLA uses termination and relevance classifiers to identify when an auxiliary objective ends and whether it applies to the high-level goal.For example, picking up a yellow key is relevant to opening a yellow door.Source: §B.4 · p. 16
  • ELLA requires expert-labelled trajectories and expert selection of auxiliary objectives, such as going to or picking up an object.Source: §B.4 · p. 17

B.5 EAGER and RIDE

The paper combines EAGER and RIDE by adding their weighted intrinsic rewards, but cannot use its usual procedure to select the weights. The combination improves Unlock-Medium results but performs worse on both PutNext tasks, making it sensitive to the chosen weights.

B.5 EAGER and RIDE

  • The method adds EAGER’s and RIDE’s intrinsic rewards with separate weights, but tuning them requires ad hoc methods because RIDE’s reward invalidates the sparsity condition used by EAGER’s weight-selection technique.Source: §B.5 · p. 17
  • The EAGER–RIDE combination improves results on Unlock-Medium but performs worse on PutNext-local and PutNext-medium, and is highly sensitive to the reward weights.The tested weights are λ_RIDE = 0.05 and λ_EAGER = 0.1; Figure 9 shows the Unlock-Medium comparison.Source: §B.5 · p. 17, Figure 9 · p. 17

C More results on EAGER

The appendix reports EAGER’s results and examines its mechanisms and scope. It includes a significant Unlock-Medium comparison, analyses of Sequence learning and QA behavior, and a commented algorithm with task-dependent reward settings.

C More results on EAGER

  • 0.8903 average return for EAGER on Unlock-Medium, versus 0.8765 for ELLA, 0.8761 for RIDE, and 0.8747 for PPO without reshaping; the EAGER–ELLA difference is significant (p = 1.9 × 10^-31).EAGER’s standard deviation is 0.011, and the authors report that it is still learning while the baselines are stuck in a local minimum.Source: C More results on EAGER · p. 17, Figure 10 · p. 18
  • 7,500 example trajectories per task were insufficient to train a successful PutNext-local policy with behavioral cloning or IQL, although IQL achieved 6% success on a GoTo test set.The paper contrasts this with at least 244,000 trajectories reported as necessary for behavioral cloning to learn PutNextTo-Local.Source: C More results on EAGER · pp. 17–18
  • EAGER’s QA achieves a 0.67 success rate on unseen Sequence-Medium goals, compared with 0.74 on goals seen during training.Source: C More results on EAGER · p. 18
  • Without neutralizing intrinsic reward on successful trajectories, the agent learns faster but ends in a local minimum because the final policy is influenced by that reward.Figure 11 compares learning with and without neutralization on PutNextTo-local.Source: C More results on EAGER · p. 18, Figure 11 · p. 18
  • After 6.5e8 frames, EAGER’s gains on Sequence goals with one or two Put Next to tasks increase total reward; these goals account for 40% and 10% of goals, respectively.The authors attribute ELLA’s slower progress on these complex goals to the time needed to learn their decomposition with its relevance classifier.Source: C More results on EAGER · p. 19, Figure 13 · p. 20
  • Across 100 checked trajectories, the QA did not guess answers from language alone; it returned no_answer when the trajectory did not support an answer.The training examples include mismatched question–trajectory pairs so the QA can learn to withhold answers until the path matches.Source: C More results on EAGER · pp. 19–20, Figure 14 · p. 20

D Commented EAGER algorithm

  • The algorithm generates questions by masking words in the goal, removes correctly answered questions from the active set, and rewards correct answers with λ times QA confidence.On successful trajectories, it subtracts discounted intrinsic rewards at the final step to avoid changing the optimal policy.Source: D Commented EAGER algorithm · p. 23

E Hyperparameters tables

  • The task-specific λ is computed using the discount factor, the maximum number of generated questions, and an assumed worst-case successful-trajectory length of N steps.The appendix states γ = 0.99 and defines the task reward at step t as r_t = 20(1 − 0.9^(t/H)).Source: E Hyperparameters tables · p. 24, Table 5 · p. 24
Loading 2206.09674v4…