PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards

• Reward functions trade design ease against learning signal: sparse rewards are easy to specify but delay feedback, while dense rewards are harder to design and tune.Source: §1 · p. 1
• PixL2R maps pixels to rewards from free-form task descriptions, avoiding assumptions about predefined object properties or structured commands.Source: §1 · p. 1
• The approach first learns trajectory–language relatedness from paired data, then uses it to generate intermediate rewards during standard RL policy training.Source: §1 · p. 1
• In the green-button example, language-generated intermediate rewards can guide an agent that otherwise explores randomly until it completes the task.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Across diverse Meta-World tasks, language-based rewards improved policy-learning sample efficiency in both sparse and hand-designed dense reward settings.Source: §1 · p. 2

• Unlike methods that infer action sequences from language, this approach generates rewards for the current state for standard RL policy learning without requiring environment dynamics.Source: §2 · p. 2
• Prior language-to-reward methods [12] assume a specific reward-function structure, whereas this approach makes no such assumption.Source: §2 · p. 2
• Instruction-following methods [15] [16] use predefined instructions or hand-designed features; this approach learns language–trajectory associations from a small set of human descriptions.Source: §2 · p. 2
• Unlike methods that ground language while interacting with the environment [18] [6], this approach uses a separate supervised phase that requires no environment interaction.Source: §2 · p. 2
• Goyal et al. [7] generate language-based rewards from actions alone and require discrete action spaces, unlike this approach.Source: §2 · p. 2
• Compared with vision-language navigation methods [26], this approach applies to continuous and discrete actions but currently handles relatively simple instructions; combining both strengths is future work.Source: §2 · p. 3

3 Approach

• The two-phase framework first trains PixL2R on paired trajectory–language data, then uses it to shape rewards for a new task.The trained model can serve downstream tasks when their objects and vocabulary closely match its training data, amortizing its training cost.Source: §3 · p. 3

3.2 Policy Learning Phase

• PixL2R scores the agent’s trajectory against a task description and uses the relatedness score to generate intermediate rewards during policy learning.The policy also receives the environment’s extrinsic reward; potential-based shaping preserves the optimal policies under the original and shaped rewards.Source: §3 · p. 3, §3.2 · pp. 4–5

3.1.1 Network Architecture

• Independent CNNs encode frame sequences from multiple viewpoints, which are concatenated and passed through a two-layer LSTM to represent the trajectory.A separate two-layer LSTM encodes the language description; the trajectory and language encodings are concatenated to predict relatedness.Source: §3.1.1 · pp. 3–4

3.1.2 Data Augmentation

• Training randomly drops frames and samples partial trajectories to reduce input size, improve robustness to minor trajectory variations, and match policy-time inputs.During policy training, trajectories are subsampled to one frame in every ten.Source: §3.1.2 · p. 4

3.1.3 Training Objectives

• Binary classification predicts RELATED or UNRELATED logits and trains with cross-entropy, while preliminary experiments found lower error on unseen complete trajectories when training used only complete trajectories.The latter comparison motivated experiments with regression on partial trajectories.Source: §3.1.3 · p. 4
• Regression assigns incomplete trajectories a length-scaled relatedness target, making incorrect-sign predictions on shorter trajectories incur smaller losses.The target is s·l/L, where s is 1 for related and −1 for unrelated pairs; mean squared error is minimized.Source: §3.1.3 · p. 4

4.2 Data Collection

• The dataset expands 13 Meta-World tasks into 1,300 scenarios, placing the task-relevant object among randomly located scene objects.Each task has 100 scenarios; 79 scenarios per task were used for training, 18 for validation, and 3 for testing.Source: §4.2 · p. 5
• PPO policies generated task videos, and Mechanical Turk workers selected or supplied English descriptions, yielding 520 descriptions, or 40 per task on average.Descriptions could be paired across the 100 scenarios for a task because most referred only to the manipulated object.Source: §4.2 · p. 5
• Negative training pairs use descriptions of alternate objects in the scene, or of other tasks when only one object is present, to test interaction rather than object presence.The dataset contains 264 unique words, with 248 in the training set; Figure 3 shows the distribution of description lengths.Source: §4.2 · p. 6, Figure 3 · p. 6

5.1 Policy Training with Language-based Rewards

• PPO was evaluated across 13 tasks and three test scenarios; comparisons focused on 16 of 39 scenarios where Dense significantly outperformed Sparse.Each policy ran for 500,000 timesteps; language-reward training used 15 runs per selected scenario.Source: §5.1 · p. 6
• Sparse+RGR was significantly better than Sparse (p=0.007), and Dense+RGR was significantly better than Dense (p=0.034); Figure 4 plots policy-training curves.The tests used normalized scenario scores and a 5% significance level.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

5.2 Ablations

• LastFrame and MeanpoolLang did not substantially reduce efficiency and remained significantly better than Dense rewards.The authors relate this to final-frame task progress and the relative simplicity of the descriptions.Source: §5.2 · pp. 7–8, Table 1 · p. 7
• MeanpoolTraj sharply reduced successful episodes and showed no significant improvement over Dense, consistent with losing movement direction.The ablation replaced the trajectory LSTM with mean-pooling.Source: §5.2 · pp. 7–8
• SingleView and Dense+CLS also lost significant improvement over Dense, suggesting multiple viewpoints and regression were useful in this setup.SingleView used one viewpoint; Dense+CLS replaced regression loss with classification loss.Source: §5.2 · pp. 7–8, Table 1 · p. 7
• Ablation performance depends on the domain; using only the last frame may hurt when the final frame does not fully reveal the state.The paper gives partially observable final-frame environments as an example.Source: §5.2 · p. 8

• PixL2R learns trajectory-language relatedness from paired data, then uses the model to generate intermediate rewards for policy learning.Source: §6 · p. 8
• In simulated robot manipulation, the approach significantly sped policy learning with both sparse and dense rewards.The authors propose using language-based rewards to improve training over coarse hand-designed dense rewards.Source: §6 · p. 8
• Proposed extensions include richer-domain generalization, sequences of instructions, attention-based alignment, and fine-tuning on policy-learning trajectories.Source: §6 · p. 8

• Figure 5 identifies the viewpoints used for data collection and experiments.Source: Figure 5 · p. 11, A Approach Details · p. 11
• Figure 6 shows three-view CNN features and language embeddings encoded by separate LSTMs, concatenated, and passed through two linear layers to predict an output.Source: Figure 6 · p. 11

• Workers saw object models alongside prototypical real-world images, encouraging real-world object names without priming specific words.Source: B Data Collection Details · p. 11
• Some responses still described models directly, such as calling the toaster model a “red box” in a “blue square.”Source: B Data Collection Details · p. 11
• Figure 7 lists the objects used in the data collection.Source: Figure 7 · p. 12

• The analysis measured each word’s influence using the mean absolute loss gradient with respect to its embedding, scaled to a maximum of 1 per description.Source: C Additional Experiments: Word-level… · p. 12
• Object words had high average gradient magnitudes across descriptions, while verbs were especially informative for faucets and windows with two possible actions.Source: C Additional Experiments: Word-level… · pp. 12–13, Table 3 · p. 13
• The analysis suggests the model identifies description words most useful for predicting trajectory-language relatedness.Source: C Additional Experiments: Word-level… · p. 13