Hybrid Reward Architecture for Reinforcement Learning

• Large real-world RL problems require strong generalisation because their state spaces are typically massive.Source: §1 · p. 1
• Deep RL methods such as DQN generalise by approximating the optimal value function with a low-dimensional representation.The value function predicts expected return from a state or state-action pair, enabling greedy policy derivation.Source: §1 · p. 1
• When the optimal value function is too complex for a low-dimensional representation, learning can become challenging or impossible.HRA therefore regularises the target side by using an easier-to-learn value function that may yield a good but non-optimal policy.Source: §1 · p. 2
• HRA decomposes the environment reward into components, trains a separate reinforcement-learning agent for each, and aggregates their action-values.The agents can learn in parallel from the same sample sequence using off-policy learning.Source: §1 · p. 2
• HRA is evaluated on a fruit-collection toy problem and the Atari game Ms. Pac-Man.Source: §1 · p. 2

• HRA builds on Horde’s parallel off-policy demons but trains separate components of the environment reward to learn a control policy.Horde instead develops general knowledge through general value functions based on separate policies and pseudo-rewards.Source: §2 · p. 2
• Unlike UVFA, which generalises across tasks or goals, HRA focuses on solving one complex task.Source: §2 · p. 2
• HRA can be viewed as applying multi-objective learning to learn a policy for a single reward function more efficiently.Source: §2 · p. 2
• Earlier reward-decomposition methods pursued optimal behaviour, whereas HRA prioritises learning efficiency by using simpler value functions and relaxing optimality requirements.Source: §2 · p. 2
• UNREAL uses auxiliary tasks to improve deep-network representations, while HRA breaks the main task into directly relevant smaller tasks.HRA is agnostic to the function approximator and can work with exact tabular representations.Source: §2 · pp. 2–3
• Diuk’s object-oriented approach efficiently learns a compact transition model and solves it with dynamic programming, but does not reduce the state space.Source: §2 · p. 3
• Options resemble HRA heads because both can train in parallel using intrinsic rewards, but options’ intrinsic rewards no longer guide the higher-level agent after training.Source: §2 · p. 3

• An MDP specifies states, actions, transitions, environment rewards, and discounting; policies select actions and aim to maximise discounted return.Source: §3 · p. 3
• DQN estimates the optimal action-value function with a neural-network approximator and minimises successive loss functions.A separate target network encodes the previous iteration’s weights.Source: §3.1 · p. 3
• A training target is consistent when its greedy policy is environment-optimal, while a semi-consistent target yields a good but non-optimal policy.Source: §3.1 · p. 3
• Consistency does not guarantee learnability: sparse or high-dimensional environment rewards can make the default optimal target difficult to learn.Source: §3.1 · pp. 3–4

Hybrid Reward Architecture

• HRA decomposes the environment reward into n components and trains a separate value function for each component.The decomposition is intended to make each value function depend mainly on a small number of state variables.Source: §3.1 · p. 4
• HRA combines the component action-values into QHRA and selects actions greedily according to their sum.The agents may share lower-level network layers and can be viewed as multiple heads of one agent.Source: §3.1 · p. 4
• The HRA heads approximate component-specific optimal action-values, while their aggregate generally differs from the environment’s optimal action-value function.Source: §3.1 · p. 4

Alternative training target

• An alternative target evaluates the uniformly random policy under each component reward, and its greedy policy can be surprisingly effective in navigation domains.This target is learned with an expected Sarsa update.Source: §3.1 · pp. 4–5

Fruit Collection task

• HRA decomposes the fruit-collection reward into 10 location-specific functions and represents each with a separate action-value head.The shared network encodes the agent’s position and fruit features, while the heads represent four action-values for each reward component.Source: §4.1 · pp. 5–6
• Domain knowledge is added incrementally by removing irrelevant features, identifying terminal states, and using pseudo-reward GVFs for each fruit location.These variants are named HRA+1, HRA+2, and HRA+3; DQN+1 uses the corresponding reduced-feature network.Source: §4.1 · p. 6
• HRA shows a clear performance boost over DQN with the same network, while adding domain knowledge produces further large improvements.Using the domain-knowledge-enhanced network improves HRA but decreases DQN performance.Source: §4.1 · p. 6

ATARI game: Ms. Pac-Man

• In Ms. Pac-Man, HRA decomposes shaped rewards by object, assigning separate agents and Q-value functions to pellets, fruits, ghosts, and blue ghosts.The architecture uses object-specific GVFs and reward weights, with absent objects contributing zero Q-values.Source: §4.2 · p. 7
• HRA’s best configuration outperforms state-of-the-art and human scores under both fixed-start and random-start evaluation after about 150 million frames.The reported configuration uses normalization and both exploration types, whereas no exploration performs very poorly.Source: §4.2 · p. 8
• With simplified executive memory, HRA reaches the maximum possible score of 999,990 points in fewer than 3,000 episodes by exploiting fixed-start evaluation.The score ceiling results from overflow at one million points, and the simplified executive memory does not generalise.Source: §4.2 · p. 8

Discussion

• Figure 6 examines training with trajectory memorisation as part of the HRA discussion.The supplied passage identifies the figure’s training condition but does not state its outcome.Source: Figure 6 · p. 9
• HRA can exploit domain knowledge more effectively than single-head methods, improving on the fruit collection task when irrelevant features are removed.The same architectural change decreased DQN performance.Source: §5 · p. 9
• HRA learns close to 1,800 general value functions for Ms. Pac-Man, replacing a state-space of about 10^77 with GVF state-spaces of about 10^3 states.These smaller problems can be represented without function approximation.Source: §5 · p. 9
• Reward decomposition can exponentially reduce problem size, so even two or three reward components may substantially help on real-world tasks.This is presented as a broader applicability claim for tasks permitting reward decomposition.Source: §5 · p. 9

General information

• Ms. Pac-Man contains four maps and seven fruit types, while gameplay rewards pellets, ghosts, power pellets, and bonus fruits.Bonus fruits appear twice per level, and completing a pellet layout starts a new level.Source: §A.1 · p. 12
• Two Atari evaluation methods produce very different scores because they introduce randomness through different starting-point procedures.The random-start metric samples starting points along a human trajectory.Source: §A.1 · p. 12
• 6.673 points is the best reported fixed-start score, while 2.251 points is the best reported random-start score among the cited methods.The cited human baseline is 15.693 points, while the highest reported human score is 266.330.Source: §A.1 · p. 12
• Figure 7 presents the four different Ms. Pac-Man maps used in the domain.Tables 2 and 3 provide map, fruit-type, and edible-object point breakdown information.Source: Figure 7 · p. 12, Table 2 · p. 13, Table 3 · p. 13
• Table 4 reports Ms. Pac-Man scores separately for fixed-start and random-start evaluation.The table labels fixed-start evaluation as random no-ops and random-start evaluation as human starts.Source: Table 4 · p. 13

GVF heads

• GVF heads create online value functions for visited Ms. Pac-Man positions, each estimating a pseudo-reward associated with its position.The Ms. Pac-Man state uses map position and heading, with roughly 400 positions and 950 states.Source: §A.2 · p. 13

Aggregator

• The aggregator activates object-specific GVFs with type-dependent multipliers and sums their action values.Multipliers include 10 for pellets, 50 for power pellets, 200 for fruits, and 1,000 for edible ghosts.Source: §A.2 · p. 13

Architecture comparison

• Figure 8 tracks training as HRA heads are added incrementally.The supplied figure passage identifies the comparison but does not report numerical outcomes.Source: Figure 8 · p. 14

Diversification head

• A diversification head samples random action values uniformly from [0,20] during the first 50 steps to make trajectories unique.The method avoids relying on ϵ-greedy exploration because unsafe actions may endanger Ms. Pac-Man.Source: §A.2 · pp. 13–14

Score heads normalisation

• Without score-head normalisation, the architecture initially reaches 10,000 points but later regresses because ghost-avoidance values overwhelm the aggregator near level completion.The resulting behavior becomes increasingly scared of ghosts as learning progresses.Source: §A.2 · p. 14

Targeted exploration head

• The targeted exploration head prioritises less-explored state-action pairs and, together with diversification, forms the HRA architecture used in every experiment.Diversification makes trajectories unique, whereas targeted exploration prioritises exploration and speeds learning.Source: §A.2 · pp. 14–15

Executive memory head

• The executive memory head records action sequences that complete a level without a kill and assigns high value to recorded actions on the same level.It models memorisation of favorable situations described in human play.Source: §A.2 · p. 15

Baseline setup

• A3C is evaluated on both pixel-based and low-level-feature environments to reproduce prior results under HRA’s input representation.The implementation follows Mnih et al. (2015).Source: §A.3 · p. 15
• A3C training uses 8.108 frames, γ = 0.99, entropy regularisation 0.01, five-step returns, 16 threads, and gradient clipping 40.α is selected for maximum performance from five tested values between 0.0001 and 0.001.Source: §A.3 · p. 15

Low-level features

• The low-feature input uses five 40 × 40 channels for Ms. Pac-Man, pellets, ghosts, blue ghosts, and fruit, plus direction information.Two convolutional layers use 16 and 32 filters with the stated kernel and subsampling settings.Source: §A.3 · p. 15

Hyperparameter evaluation

• Figure 9 compares smoothed training curves for γ values without executive memory.The figure is smoothed over 500 episodes.Source: Figure 9 · p. 15
• Figure 10 reports a γ grid search with executive memory.The supplied passage does not state the comparison outcome.Source: Figure 10 · p. 16

Network architecture

• The baseline network has a 256-unit hidden layer, a nine-unit softmax policy head, and a one-unit linear value head.Weights are uniformly initialised using the cited He et al. method.Source: §A.3 · p. 16

• All tested γ values yielded very good results, and these results generalised over random human starts.The tested values were 0.95, 0.97, and 0.99 for both γscore and γghosts.Source: §A.4 · p. 16
• Higher γ values for the ghosts tended to perform better, while the γ value for the score was less impactful.Source: §A.4 · p. 16
• The same conclusions held when executive memory was used.Source: §A.4 · p. 16
• Executive memory was associated with somewhat more randomness in level-passing efficiency.Source: §A.4 · p. 16