PaperlayerView exact revision on arXiv

Source-linked AI summary

What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization

Julia Barth, Armin Sommer, Kunvar Thaman, Jannik Schilling

arXiv:2610.19731v2cs.LGcs.AIcs.RO

TL;DR

Goal misgeneralization can remain hidden while agents perform competently, and the intended goal’s internal role and recoverability remain poorly understood. The paper traces goal information and policy behavior across visual and simulated robot tasks, finding that the depth where the goal is lost predicts which recovery works.

  • Problem

    The paper asks what happens to the intended goal inside misgeneralizing agents, a gap in work that has mainly characterized the behavior and whose proxy-following can stay hidden while agents remain competent.

    Source: §1 · p. 1

  • Method

    The authors trace goal representation, policy readout, behavior, and test-time recovery across eight environments, including simulated arm and quadruped tasks.

    Source: §1 · p. 2

  • Results

    Across the environments, recovery depends on goal-loss depth; in ATTRIBUTE MAZE, steering raises GM agents’ goal reach from 0.160 to 0.953.

    Source: §1 · p. 2, §4.1 · p. 8, §4.2 · p. 10

  • Takeaways & Limitations

    Behavior alone cannot distinguish these failures; locating where the goal is lost indicates whether steering, a new readout, or neither can recover goal use.

    Source: §5 · p. 10

    The proposed link between training-environment structure and how the goal fails is presented as a hypothesis.

    Source: §5 · p. 10

Abstract

from arXiv · show

Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but pursues a proxy that coincided with the intended goal during training instead of the goal itself. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents and what enables them to be recovered without retraining. We study this across eight visual navigation and manipulation tasks, including robot control. While tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, we uncover that the goal is lost at one of three different depths. This depth predicts which recovery works in each task. When the goal is followed, but outcompeted by the proxy, steering the existing policy redirects it to the goal's location, raising goal reach from 16.0% to 95.3%. When it is read, but not followed, steering fails, but a lightweight new downstream wiring recovers goal reach from 31.6% to 84.3%. When it is not read, none of the methods work, as the policy does not use the goal information. The same tracing also reveals what replaces the goal: functional target substitution, where behavior follows an alternative goal, a policy shortcut, where behavior follows a learned action strategy, or cue competition between intended and proxy goal cues. Knowing where the goal is lost tells us whether an agent can be repaired or has to be retrained. Goal misgeneralization should therefore be studied and evaluated inside the agent, not only from its behavior.

1 INTRODUCTION

Goal misgeneralization can remain hidden while agents perform competently because a proxy and the intended goal earn the same reward during training. The paper traces where the goal is lost inside misgeneralizing agents and relates that depth to which test-time recovery works.

1 INTRODUCTION

  • Goal misgeneralization is dangerous because competent behavior can conceal proxy-following until the goal and proxy stop co-occurring.During training, pursuing either can earn the same reward.Source: §1 · p. 1
  • Prior work largely characterized misgeneralization behaviorally, while interpretability studied other questions and mitigation attempts intervened during training rather than afterward.The paper asks what happens to the intended goal inside misgeneralizing agents.Source: §1 · p. 1
  • Three distinct alignment failures arise at different goal-loss depths, and that depth predicts whether steering, a new readout, or neither can recover goal-directed behavior.In MAZE the goal is inert; in PUSHTOMARKER and COIN it is read but not followed; in ATTRIBUTE MAZE it is followed but outcompeted by the proxy.Source: §1 · p. 2
  • The study traces goal information and policy behavior across eight environments, including five simulated robot tasks, and characterizes cue and object competition internally.The authors also propose policy Fisher information to isolate a policy-directed view of internal representations.Source: §1 · p. 2

2 SETUP

The paper formalizes goal misgeneralization as proxy-following when a training correlation between the goal and proxy breaks at evaluation. It studies eight visual environments and analyzes both spatial feature representations and the downstream policy representation.

2 SETUP

  • Goal reach is the fraction of evaluation rollouts that reach the intended goal, while misgeneralization occurs when agents follow a training-coincident proxy after the distributions shift.The setup models each environment as an episodic Markov decision process.Source: §2 · p. 3
  • Eight visual environments pair visible goals and proxies during training, spanning Procgen games and simulated arm and quadruped tasks.Examples include MAZE, COIN, ATTRIBUTE MAZE, and PUSHTOMARKER; four further robot tasks replicate failures.Source: §2 · p. 3
  • The analysis probes hpre, the last representation retaining scene layout, and hfc, the fully connected representation read by the policy and value heads.Procgen agents use IMPALA architectures and robot agents use Nature CNNs, trained online with PPO.Source: §2 · p. 3

3 WHAT HAPPENS TO THE INTENDED GOAL?

The intended goal can be absent, read without directing behavior, or followed but outcompeted; these depths accompany distinct ways the policy organizes behavior. In ATTRIBUTE MAZE, goal and proxy cues oppose one another along a shared policy-relevant direction.

3.1 IS THE GOAL STILL THERE?

  • The goal is not linearly decodable beyond random-network levels in MAZE and PUSHTOMARKER, but remains decodable in COIN and ATTRIBUTE MAZE.In COIN, coin bearing has R2 0.378 for GM agents at hpre; in ATTRIBUTE MAZE, goal-cue AUC is 0.770 at h(2).Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 DOES THE POLICY READ OUT THE INTENDED GOAL?

  • The policy barely reacts to goal edits in MAZE (KL ratio 0.009), reacts without goal-directed follow in PUSHTOMARKER and COIN, and follows the goal in ATTRIBUTE MAZE (signed-follow ratio 0.533).The four other robot tasks fall into the same three depths.Source: §3.2 · p. 5, Table 1 · p. 5

3.3 WHAT ORGANIZES THE BEHAVIOR INSTEAD?

  • MAZE agents follow a new top-right extreme rather than the cheese, while COIN agents keep moving right past the former endpoint, indicating target substitution versus a policy shortcut.MAZE reaches a new extreme placed away from the screen corner first in 0.792 of rollouts; COIN agents continue right in 0.938 of episodes that reach the former endpoint.Source: §3.3 · pp. 5–6, Figure 3 · p. 6
  • In PUSHTOMARKER, camera shifts move GM agents’ push location; in ATTRIBUTE MAZE, both shape and colour swaps push support toward their candidate and pull it from the competitor.Only the camera shift moves where GM agents push; in ATTRIBUTE MAZE, the cues’ policy effects counteract.Source: §3.3 · p. 6, Figure 3 · p. 6

3.4 WHEN GOAL AND PROXY BOTH REACH THE POLICY, HOW DO THEY COMPETE?

  • In ATTRIBUTE MAZE, goal and proxy edits oppose each other in GM agents (Fisher cosine −0.919), and their competition directions align across same-conflict states (0.942).The colour cue pulls 1.564 times as hard as the shape cue in GM agents, versus 0.067 in controls.Source: §3.4 · p. 7, Figure 4 · p. 7

4 CAN THE GOAL BE RECOVERED?

Recovery depends on where the goal is lost: steering works when the policy already follows the goal, while new wiring can recover goal use when information still reaches the representation. Across the eight environments, these tracing results predict which repair works; neither method works when the goal is inert to the policy.

4.1 CAN THE GOAL BE RECOVERED THROUGH THE EXISTING WIRING?

  • Steering compares state-dependent corrective, reversed and random pushes at hfc, with shared-axis variants added for ATTRIBUTE MAZE.The goal–proxy counterfactual difference defines the steering direction, which is rebuilt as the agent moves; controls include reversed, equal-norm random, and unsteered policies.Source: §4.1 · p. 8, Figure 5 · p. 8
  • Goal reach in ATTRIBUTE MAZE rose from 0.160 to 0.953 at β = 1 under steering, while steering did not move GM agents toward the goal in MAZE, PUSHTOMARKER or COIN.In ATTRIBUTE MAZE, reversing the push reduced goal reach and a random push left it unchanged.Source: §4.1 · p. 8, Figure 5 · p. 8

4.2 CAN THE GOAL BE RECOVERED THROUGH NEW WIRING?

  • GM rewirings from hpre reached the coin in 0.843 of COIN episodes, versus 0.316 for native GM agents and 0.888 for the control rewire.The random-trunk rewire selected the coin over the level end in 0.962 of episodes ending at either, close to 0.987 for the GM rewire, but more often died or timed out.Source: Figure 6 · p. 9, §4.2 · pp. 9–10
  • New wiring relearns the downstream policy head or the hpre-to-hfc mapping by behavioral cloning, using oracle targets in MAZE and ATTRIBUTE MAZE and control-agent targets in COIN and PUSHTOMARKER.The comparisons include rewires of control agents, native reach, a random-trunk floor, and a fixed random remix.Source: §4.2 · p. 9, Figure 6 · p. 9
  • In MAZE, GM rewires from hpre and hfc reached the cheese in 0.116 and 0.119 of episodes, and relabeling left their reach below 0.13.The control rewire was also below native control reach after cloning, but relabeling restored it.Source: §4.2 · p. 9, Figure 6 · p. 9
  • In PUSHTOMARKER, an hpre rewire reached the marker in 0.314 of conflict scenes, rising to 0.661 with on-policy relabeling, versus 0.296 with cloning alone.The control rewire reached 0.702, while the GM rewire from hfc reached 0.132.Source: §4.2 · p. 9, Figure 6 · p. 9
  • In ATTRIBUTE MAZE, an hpre rewire reached the goal in 0.871 of episodes, near the control rewire’s 0.908 and above native GM reach of 0.358.A random trunk rewired in the same way reached 0.840; from hfc, the GM rewire reached 0.478.Source: Figure 6 · p. 9, §4.2 · p. 10
  • Across all eight environments, steering worked only where the goal was followed, and a new readout worked only where goal information still reached the policy.Recovery from hfc was weaker almost everywhere, and no method recovered goal orientation in MAZE.Source: §4.1 · p. 9, §4.2 · p. 10

5 DISCUSSION AND CONCLUSION

The paper argues that goal misgeneralization comprises distinct failures, and that diagnosing where the goal is lost helps determine whether recovery is possible. Its environments separate goal and proxy cues through locations, movement strategies, object attributes, and competing landmarks.

5 DISCUSSION AND CONCLUSION

  • The depth at which the goal is lost determines which recovery works: steering can redirect a policy when a competing cue or object overpowers a still-wired goal, while new readout can partly restore goal use when it remains represented.Neither method works when the goal is inert to the policy; behavior alone cannot distinguish these failures.Source: §5 · p. 10
  • The authors hypothesize that training-environment structure shapes failure depth, but note that they studied static checkpoints and only part of goal misgeneralization.They leave open how the goal is lost during training and how failures differ in settings such as instrumental subgoals and language models.Source: §5 · p. 10

A ENVIRONMENTS

  • The eight environments pair an intended goal with a proxy that co-locates during misgeneralization training, then separate them in conflict tests to measure which the agent pursues.The set includes three Procgen games, four simulated arm tasks, and one quadruped navigation task.Source: A ENVIRONMENTS · pp. 17–18

A.1.1 MAZE: DESTINATION AMBIGUITY

  • In MAZE, the cheese’s fixed top-right training location permits a corner shortcut; at conflict test the cheese moves to another cell.The corner proxy is scored as entering the top-right region without reaching the cheese.Source: §A.1.1 · p. 18

A.1.2 COIN: STRATEGY AMBIGUITY

  • In COIN, the coin’s training position at the level end makes persistent rightward movement a reliable proxy strategy; conflict levels place the coin in a random section.Reaching the end wall without collecting the coin counts as proxy behavior.Source: §A.1.2 · p. 19

A.1.3 ATTRIBUTE MAZE: CUE AMBIGUITY

  • In ATTRIBUTE MAZE, training pairs the rewarded yellow line’s shape and colour cues, while the conflict test splits them between a red line and a yellow gem.The conflict is new to both cohorts, and the objects are equally easy to reach.Source: §A.1.3 · pp. 20–22

A.2 ROBOT-ARM ENVIRONMENTS

  • In the arm tasks, proxies include the table-centre destination in PUSHTOMARKER and fixed training corner in LOCATIONREACH; conflict tests move the goal away from these locations.ATTRIBUTETABLE and SORTZONES instead test colour proxies while keeping object positions or push requirements consistent across distributions.Source: §A.2.1 · p. 22, §A.2.2 · p. 22, §A.2.3 · p. 23

A.2.3 ATTRIBUTETABLE: ATTRIBUTE AMBIGUITY IN A TOUCH

  • ATTRIBUTETABLE agents still touched the cube first in 0.921 [0.824, 1.017] of episodes when both objects were red, indicating shape decided when colour did not.In the conflict test, the cube is blue and the sphere red, with positions drawn identically across distributions.Source: §A.2.3 · p. 23

A.3.1 WALKTOFLAG: A LANDMARK BESIDE THE GOAL

WALKTOFLAG contrasts a small green flag with a much more visible orange tower beside it. The conflict test separates the two targets without introducing a new walking direction.

A.3.1 WALKTOFLAG: A LANDMARK BESIDE THE GOAL

  • The quadruped follows a frozen pretrained walking controller’s velocity and turn-rate commands in a walled 10 × 10 m arena.The policy uses a Nature CNN and is trained with PPO.Source: §A.3.1 · p. 23, Table 3 · p. 24
  • The orange tower is a far more visible proxy than the green flag, and removing it lets GM agents reach the flag first in 0.984 [0.959, 1.010] of episodes when both objects are in view.The flag banner covers 1 to 2 faint pixels at 3 m, while the tower covers 290; the conflict places the tower away from the direct route to the flag.Source: §A.3.1 · p. 23
  • The conflict test ends the episode at the first object reached, within 0.5 m of the flag or 0.6 m of the tower; falls and timeouts count as neither.Flag and tower occur across the arena in every distribution, so no walking direction is new at test.Source: §A.3.1 · p. 23

B FORMAL SETUP AND NOTATION

The formal setup distinguishes task-level goal achievement from reward and proxy regularities, and separates information represented internally from how the trained policy uses it. It defines tools for analyzing controlled world edits, decision states, representation projections, and task-specific target geometry, with an extended analysis of three goal-failure patterns across eight environments.

B FORMAL SETUP AND NOTATION

  • Rendered observations serve as policy inputs, and intended-goal success is reaching a goal-satisfying state, measured as its probability across procedurally generated levels.The setup models an episodic Markov decision process and keeps the intended goal distinct from the policy input representation.Source: B FORMAL SETUP AND NOTATION · p. 25
  • Reward is the learning signal rather than the task-level goal: Procgen reward is sparse, while robot reward is dense and also pays for progress.In both settings, reward indicates the intended goal without being identical to it.Source: B FORMAL SETUP AND NOTATION · p. 25
  • Training-correlated proxy regularities may be alternative goals, object attributes, or directions; OOD evaluation breaks their training association so intended-goal and proxy-following behavior diverge.The proxy need not itself be a separate goal, and successful training trajectories make it strongly predictable.Source: B FORMAL SETUP AND NOTATION · pp. 25–26

B.1 NETWORK ARCHITECTURE

  • Procgen agents use three spatial IMPALA blocks before hpre (2048 dimensions) and hfc (256); robot agents use a Nature CNN, and policy and value heads read hfc.The Procgen shared actor-critic architecture is illustrated in Figure 12.Source: §B.1 · p. 27, Figure 12 · p. 27
  • hpre retains spatial structure, whereas hfc is the vector representation read by the heads; comparing them separates spatially carried information from information passed to the readout.At hfc, policy changes pass through the linear policy head.Source: §B.1 · p. 27

B.2 THEORY: LOCAL POLICY GEOMETRY

  • Probing tests whether goal-related information is linearly available, not whether the policy responds; activation-space Fisher measures local sensitivity through the particular downstream policy.Directions with high Fisher sensitivity strongly affect the policy distribution, while low-sensitivity directions are comparatively ignored.Source: §B.2 · p. 28
  • World-edit Fisher sensitivity combines an edit-induced representation displacement with downstream Fisher geometry to approximate local policy change, while behavioral KL measures the complete finite change.A representation displacement can be large or linearly informative yet have small Fisher sensitivity if the downstream computation is insensitive to its direction.Source: §B.2 · pp. 29–30
  • For robot agents' Gaussian policies with state-independent noise, activation-space Fisher depends on the action-mean Jacobian; the softmax-specific closed form does not apply.The appendix also gives an exact finite KL expression for the Gaussian policy between base and edited states.Source: §B.2 · p. 31
  • Small Fisher sensitivity Q does not show that edit-related information is absent; it measures how the trained downstream policy reads the representation.Agents can encode the same information yet have different Fisher geometries because their downstream computations use it differently.Source: §B.2 · p. 32
  • The Fisher decomposition separates representation displacement from directional policy sensitivity, while Q, G, and DF remain invariant to scalar rescaling that preserves the policy.A change of representation site can alter G and DF, so the paper compares them across cohorts only at the same site.Source: §B.2 · pp. 32–33
  • 82–86% of real state pairs exceed |cosF| > 0.9 across the three black-background GM policies, versus essentially no matched random pairs, so low Fisher rank alone does not explain near-±1 structure.The comparison uses 256-dimensional random vectors and rank-14 random Fisher metrics.Source: §B.2 · p. 34
  • The signed target-selection derivative tests whether an edit-induced representation change favors intended- or proxy-consistent actions, complementing Q’s measure of overall policy sensitivity.Its raw form combines edit size and direction; the normalized form measures target-selective sensitivity per unit displacement.Source: §B.2 · p. 35
  • Policy-Fisher input saliency measures the whole action distribution, weights actions by their probabilities, and is invariant to a common shift of all logits.Because high-contrast sprites attract saliency regardless of learned behavior, regions are compared with randomly initialized networks and at fixed distance from the agent; saliency maps suggest hypotheses, while causal claims rely on edits and rollouts.Source: §B.2 · pp. 35–36

B.3 PROJECTIONS AND VECTOR ANALYSIS

  • Euclidean projection measures activation-space alignment, whereas Fisher-orthogonal projection isolates the policy-sensitive component and yields an exact additive Fisher quadratic decomposition.Euclidean orthogonality does not generally imply Fisher orthogonality.Source: §B.3 · pp. 36–37
  • The signed-effect decomposition is exactly additive for the local directional derivative, but this does not imply additivity for arbitrary finite displacements.Source: §B.3 · p. 37
  • A shared-axis approximation represents the relevant displacement component with a fixed direction and state-dependent sign and magnitude, without asserting that the full representation geometry is one-dimensional.The residual retains remaining state-specific structure.Source: §B.3 · p. 37
  • The intervention families compare full local directions, gated shared axes, unsigned shared axes, and Fisher-projected components to test alternative compression hypotheses.Source: §B.3 · p. 38

B.4 ENVIRONMENT-SPECIFIC TARGET GEOMETRY

  • On training states, intended and proxy goals are often action-confounded; graph-exact progress sets account for walls and identify actions that strictly reduce shortest-path distance to each target.These action sets guide both evaluation-state selection and behavioral scoring.Source: §B.4 · p. 39
  • Evaluation uses conflict states where intended and proxy targets imply different immediate action sets, then restricts banks to strict conflicts so every progress action serves exactly one target.ATTRIBUTE MAZE additionally requires candidate targets to be equidistant, avoiding a proximity advantage.Source: §B.4 · p. 40
  • Robot target directions use task-plane motion from the mean action; quadrupeds use planar velocity commands, while arm-task motion maps joint changes through the Jacobian to object centers in reach and table tasks.The planned step is computed from the unclipped mean action.Source: §B.4 · p. 41
  • In PUSHTOMARKER and SORTZONES, the target point is 3 cm behind the cube center on the line from the goal through the cube, where the gripper must stand to push it toward the goal.This point is 1 cm behind the cube’s rear face and is also used by the dense reward.Source: §B.4 · p. 41

C EXPERIMENTAL DESIGN AND EXTENDED RESULTS

  • The analysis traces goal information from representation availability to policy readout and proxy organization, then tests recovery by steering or learning a new readout.The recovery tests aim to restore goal-directed behavior without further reinforcement learning.Source: C EXPERIMENTAL DESIGN AND EXTENDED… · p. 42
  • Across eight environments, target substitution, policy shortcuts, and cue competition replace or compete with the goal, while goal-survival depth aligns with recovery type.Target substitution occurs in MAZE, LOCATIONREACH, and PUSHTOMARKER; policy shortcut in COIN; and cue competition in ATTRIBUTE MAZE, ATTRIBUTETABLE, SORTZONES, and WALKTOFLAG.Source: Figure 13 · p. 42, C EXPERIMENTAL DESIGN AND EXTENDED… · p. 42

C.1 LINEAR DECODABILITY: IS INFORMATION ABOUT THE INTENDED GOAL STILL THERE?

This section examines how accessible intended-goal information is across eight environments and distinguishes its availability from whether the policy follows it, from information that fails to reach the policy to information that is followed but outcompeted by the proxy.

C.1 LINEAR DECODABILITY: IS INFORMATION ABOUT THE INTENDED GOAL STILL THERE?

  • MAZE GM bearing probes collapsed when agent position was decorrelated from cheese bearing, whereas COIN bearing remained decodable beyond position in both cohorts despite weaker GM coin-presence representation.The MAZE result indicates that the GM bearing signal largely tracked agent position; COIN bearing retained information beyond position through hpre, while coin presence became nearly ceiling-level only in Non-GM agents.Source: Figure 14 · p. 44, §C.1 · pp. 44–46, Figure 15 · p. 45
  • Where proxies are attributes or related landmarks, hfc representations generally favor the proxy in GM agents and the goal in Non-GM agents, though ATTRIBUTE MAZE shows this only partly.SORTZONES supports this contrast reliably only at hfc because earlier layers have little headroom above random networks; ATTRIBUTE MAZE cue accessibility does not consistently separate the cohorts.Source: §C.1 · pp. 47–49, Table 5 · p. 49
  • Linear decodability shows accessible information, not whether the policy uses it as a goal or can act on it; baseline headroom also limits what probes establish.For example, SORTZONES random features decode crossed-pad position at hpre, but a behavior-cloned readout on those features does not recover successful goal-directed behavior.Source: §C.1 · p. 49
  • Across tasks, goal information was not uniformly erased: it remained linearly accessible in many GM agents, while MAZE, LOCATIONREACH, and PUSHTOMARKER lacked above-random goal decoding.ATTRIBUTE MAZE, ATTRIBUTETABLE, SORTZONES, and WALKTOFLAG showed goal information reaching late layers, often with the proxy stronger in GM agents and the goal stronger in Non-GM agents; COIN was intermediate.Source: §C.1 · p. 50

C.2 BEHAVIORAL DEPENDENCE: DOES THE POLICY REACT TO GOAL CHANGES, VIEWED FROM OUTSIDE?

  • The behavioral-dependence analysis edits goal- or proxy-related variables while holding the rest of the state fixed where possible, then tests whether policy output follows the edited feature.Relocation edits move objects such as the cheese, coin, marker, ball, flag, or tower; the supplied passages describe the intervention design but not its results.Source: §C.2 · p. 50
  • Behavioral dependence is measured first by edit-induced action-distribution KL and then by signed follow, which tests whether probability shifts toward actions advancing toward the edited goal or proxy.For robot policies, which already place more than 0.999 probability in the progress half-space in ATTRIBUTETABLE, the measure instead tracks how the edit turns the planned step.Source: §C.2 · p. 52
  • Goal edits barely change GM policies in MAZE (KL ratio 0.009) and LOCATIONREACH (0.026), while ratios in the other six environments range from 0.107 to 0.362.In the three swap environments and WALKTOFLAG, both goal and proxy edits move the policy.Source: §C.2 · p. 53, Figure 19 · p. 54
  • Goal-edit signed-follow ratios are near zero in PUSHTOMARKER (0.017) and COIN (0.001), but range from 0.234 to 0.533 in WALKTOFLAG and the three swap environments.In PUSHTOMARKER and COIN, edits can change action probabilities without shifting them toward goal-directed actions; proxy edits instead turn GM policies several times more than Non-GM policies.Source: §C.2 · p. 54, Figure 19 · p. 54
  • The combined metrics distinguish goal edits that do not reach the policy in MAZE and LOCATIONREACH, reach but are not followed in PUSHTOMARKER and COIN, or are followed but lose to the proxy elsewhere.The latter environments are WALKTOFLAG and the swap environments.Source: §C.2 · p. 54

C.3 FISHER GEOMETRY: IS THE POLICY USING THE GOAL, VIEWED FROM INSIDE?

Fisher sensitivity reveals that misgeneralized policies relate to goal edits in distinct ways: they may barely react, react without moving toward the goal, or read the goal alongside a stronger proxy. Across these tasks, the same analyses connect where the goal reaches the policy with how behavior is organized over an episode.

C.3 FISHER GEOMETRY: IS THE POLICY USING THE GOAL, VIEWED FROM INSIDE?

  • World-edit Fisher sensitivity Q measures how strongly an edit changes the action distribution, while signed target-selection effect indicates whether the shift favors the goal or proxy.Q is unsigned, and a small Q does not establish that information about the edited property is absent.Source: §C.3 · p. 55
  • In MAZE and LOCATIONREACH, goal edits reach GM policies far less than Non-GM policies, with Q ratios below 0.001 and 0.023, respectively.In MAZE, the small displacement indicates goal loss before the policy head; in LOCATIONREACH, both displacement and directional gain contribute.Source: §C.3 · pp. 58–59
  • In PUSHTOMARKER and COIN, goal edits reach GM policies at Q ratios of 0.159 and 0.063, but the reactions do not direct action toward the goal.The planned GM gripper step does not turn toward the moved marker, and the COIN signed effect remains near zero.Source: §C.3 · p. 58
  • In ATTRIBUTE MAZE, ATTRIBUTETABLE, SORTZONES, and WALKTOFLAG, proxy edits outweigh goal edits within GM policies, while goal edits still affect them.The proxy-edit to goal-edit Q ratios are 1.938, 7.359, 10.565, and 18.080, respectively.Source: §C.3 · p. 58
  • The decomposition Q = G DF distinguishes small representation displacement from displacement off policy-read directions, but G and DF are descriptions rather than decision rules.The decomposition depends on site coordinates, and wide intervals from three robot seeds mean its ordering is indicative.Source: §C.3 · p. 59
  • Across the environments, goal edits barely move the native policy in MAZE and LOCATIONREACH, move it without directing behavior in PUSHTOMARKER and COIN, and remain coupled alongside stronger proxy edits in four other tasks.These inside-policy patterns mirror behavioral analyses; the common label “goal misgeneralization” therefore covers different relationships between goal and policy.Source: §C.3 · pp. 58–59

C.4 WHAT ROLE DOES THE INTENDED-GOAL INFORMATION PLAY?

  • The episode-level analyses distinguish functional target substitution, a policy shortcut, and cue or object competition as ways behavior can be organized when goal information remains.The chapter groups five environments by the substitute target and the policy’s relation to the goal.Source: §C.4 · p. 60

C.4.1 FUNCTIONAL TARGET SUBSTITUTION: MAZE AND LOCATIONREACH

  • GM agents pursue the top-right maze extreme and LOCATIONREACH corner as targets; in LOCATIONREACH the target follows the camera image rather than the arm base.MAZE agents follow a newly created top-right extreme, while LOCATIONREACH agents reach the corner on 0.993 of off-path episodes and the pixel-implied point on 0.993 after a camera shift.Source: §C.4.1 · p. 62

C.4.2 TARGET SUBSTITUTION WITH UNDIRECTED GOAL WIRING: PUSHTOMARKER

This section examines how behavior may be organized by substitute targets, learned action rules, or competition between goal and proxy cues. It covers tests of object competition in WALKTOFLAG and the geometry of cue edits within and across states.

C.4.2 TARGET SUBSTITUTION WITH UNDIRECTED GOAL WIRING: PUSHTOMARKER

  • On separable layouts, GM agents push the cube to the centre in 0.980 of episodes and onto the marker in none, while Non-GM agents reach the marker in 0.935.With the camera shifted by 5 cm, GM agents push the cube to the point projecting to the old centre pixel in 0.901 of episodes and to the real centre in 0.018. Drawing or hiding the marker moves GM paths by 0.050 cm, versus 10.504 cm for Non-GM agents.Source: §C.4.2 · pp. 64–65

C.4.3 POLICY SHORTCUT: COIN

  • GM policies continue past the former endpoint in 47 of 50 episodes that reach it (0.938 per checkpoint), supporting an action-rule shortcut rather than stopping at a substitute target.They spend 3% of post-arrival steps at the former wall face, versus 100% with the wall in place; every continuing agent stops at the next wall, so the test does not rule out running toward a visible end wall.Source: §C.4.3 · pp. 66–67
  • Relocating the coin changes GM policy output (total variation 0.080 versus 0.280 for Non-GM) but adds +0.000 probability to coin-directed actions, versus +0.179 for Non-GM.The GM action-share ratio is 1.210, near the nondirectional reference of 0.991 for randomly initialized networks; across distances, GM coin-directed action changes remain between −0.001 and 0.003. Saliency supports but does not prove this reading: GM coin saliency is 2.13× the image mean, near random networks at 2.78×.Source: §C.4.3 · p. 67, Figure 25 · p. 67

C.4.4 CUE COMPETITION: ATTRIBUTE MAZE, ATTRIBUTETABLE AND SORTZONES

  • In ATTRIBUTE MAZE, ATTRIBUTETABLE, and SORTZONES, both matched cue swaps reorganize GM target preference, while Non-GM policies are controlled by the goal cue.GM agents favor the proxy cue: in ATTRIBUTE MAZE on black, the colour swap shifts preference by 3.089 versus 4.355 for the shape swap; the swaps restore familiar conjunctions, so their effects do not identify independent internal cue channels or weights.Source: Figure 27 · p. 69, §C.4.4 · pp. 70–71, Table 12 · p. 70

C.4.5 OBJECT COMPETITION: WALKTOFLAG

  • WALKTOFLAG tests whether the flag and a more visible tower each pull the agent, and whether competition differs from cue competition within one object.The design removes each object to measure its effect on walking commands, uses matched swaps, and compares object pulls against the distance gap to test whether behavior simply follows the nearer object; no outcome is reported in these supplied passages.Source: §C.4.5 · pp. 71–72
  • In WALKTOFLAG, the tower pull is 3.736 policy standard deviations versus 0.211 for the flag, although both objects can pull simultaneously without pushing the agent away from the other.The tower dominates at every distance, while the flag’s pull is weak and persists mainly when it is near.Source: §C.4.5 · p. 72, Figure 28 · p. 73

C.5.1 HOW ARE THE TWO CUE DIRECTIONS ORGANISED WITHIN A STATE?

  • Within-state Fisher cosine cosF is −0.919 in ATTRIBUTE MAZE, −0.435 in SORTZONES, −0.160 in WALKTOFLAG, and −0.002 in ATTRIBUTETABLE, showing opposition varies by task.In ATTRIBUTETABLE, opposed gripper motions are partly offset by aligned policy changes in arm-joint directions that leave the gripper in place; in SORTZONES, both components are opposed.Source: Figure 29 · p. 74, §C.5.1 · pp. 74–75

C.5.2 IS THE COMPETITION DIRECTION SHARED ACROSS STATES?

  • Held-out competition directions align with a shared axis at cosF = 0.959 in ATTRIBUTE MAZE; GM agents also gain κF above initialization by 0.599 in ATTRIBUTETABLE, 0.249 in SORTZONES, and 0.213 in WALKTOFLAG.The axis is not specific to goal misgeneralization: Non-GM agents share it almost as much. Results vary with the state bank; for example, ATTRIBUTETABLE same-sign κF is 0.281 on the world-edit bank versus 0.667 and 0.884 on the strict bank.Source: Table 13 · p. 76, §C.5.2 · pp. 76–78
  • In ATTRIBUTE MAZE, single-cue ablations recover the swap-axis sign in 92% to 100% of held-out states, supporting that the axis reflects trained policies rather than the swap edit.On a diversified bank, the single axis splits into roughly one shared axis per action-conflict class: κF = 0.889 for pairs sharing the action contrast and bearing, versus −0.213 for pairs sharing neither.Source: §C.5.2 · p. 78

C.5.3 WHY DOES THE PROXY WIN ON THE SHARED AXIS?

On the shared competition axis, the proxy pulls harder than the goal in every GM cohort, while the geometry and behavioral consequences vary by environment.

C.5.3 WHY DOES THE PROXY WIN ON THE SHARED AXIS?

  • Median proxy-to-goal pull ratios are 1.527 (GM) vs 0.082 (Non-GM) in ATTRIBUTE MAZE, and 5.106/6.216/4.228 vs 0.267/0.211/0.449 across ATTRIBUTETABLE/SORTZONES/WALKTOFLAG.The robot-task values are ordered ATTRIBUTETABLE, SORTZONES, WALKTOFLAG.Source: §C.5.3 · p. 78, Figure 32 · p. 79
  • Initialized networks also favor the proxy on average, but in robot tasks this advantage is mostly an input effect that GM training preserves while Non-GM training reverses.In the robot tasks, the tower occupies about 14 times more pixels than the flag; per unit of pixel change, initialized networks do not favor the proxy.Source: §C.5.3 · pp. 78–79
  • Both cues pull toward their own candidates in 0.967 of GM ATTRIBUTE MAZE states, versus 0.884 in ATTRIBUTETABLE, 0.820 in SORTZONES, and 0.650 in WALKTOFLAG.The tug is balanced in ATTRIBUTE MAZE, lopsided in ATTRIBUTETABLE and SORTZONES, and inconsistent in WALKTOFLAG.Source: §C.5.3 · p. 79
  • The proxy pull alone predicts GM outcomes about as well as both pulls together (AUC 0.779), while the goal pull alone is near chance (0.459).Across 196 ATTRIBUTE MAZE states, signed distance from the midpoint predicts outcomes with AUC 0.774; the analysis was not run in the robot tasks.Source: §C.5.3 · p. 80, Figure 33 · p. 80

C.5.4 WHAT DOES THE GEOMETRY SUGGEST FOR RECOVERING THE GOAL?

The competition geometry suggests which steering strategy may work, and the authors then frame recovery as a test of whether existing internal goal information can redirect behavior.

C.5.4 WHAT DOES THE GEOMETRY SUGGEST FOR RECOVERING THE GOAL?

  • ATTRIBUTE MAZE has the clearest shared-axis tug: cues are nearly opposed, align on one axis, and pull their own way in almost every state, making a fixed signed push a promising recovery direction.Its shared-axis measure κF is 0.793 above initialization.Source: §C.5.4 · p. 80
  • The other tasks complicate shared-axis steering: WALKTOFLAG lacks a strong global axis, SORTZONES has a weak one, and ATTRIBUTETABLE's aligned joint component may require a different or larger push.In SORTZONES, state-specific steering may still work; ATTRIBUTETABLE's shared axis moves the gripper contrast but leaves the aligned component in place.Source: §C.5.4 · pp. 80–81

D RECOVERABILITY: CAN GOAL-DIRECTED BEHAVIOR BE RECOVERED?

  • The proposed recovery test asks whether steering the existing policy or rewiring downstream of a representation can use internal goal information, with the depth of goal loss hypothesized to predict which works.The authors expect steering to suffice when goal loss occurs at the decision, rewiring when information reaches the policy but is unused, and neither method when it is absent from the representation.Source: D RECOVERABILITY: CAN GOAL-DIRECTED… · p. 82

D.1 STEERING THE EXISTING WIRING

  • Steering tests whether identified internal geometry can redirect the existing policy without retraining its weights; the interventions are applied at hfc, which the policy and value heads read directly.In ATTRIBUTE MAZE, pushing at hpre yields goal reach within 0.06 of the hfc push at every tested dose up to β = 1.Source: §D.1 · p. 82
  • The steering analysis separates the causal effect of each state's counterfactual direction from the effect retained when that direction is compressed into a shared axis.These are the two questions posed for environments where local steering succeeds.Source: §D.1 · p. 82

D.1.1 CAN THE STATE-LOCAL COUNTERFACTUAL DIRECTION REDIRECT THE POLICY?

State-local steering recovers goal-location reach when goal and proxy compete, but fails when the goal is absent from the representation or reaches the policy without being used. The authors interpret successful steering as location redirection, not necessarily restored goal-cue pursuit.

D.1.1 CAN THE STATE-LOCAL COUNTERFACTUAL DIRECTION REDIRECT THE POLICY?

  • The intervention uses a state-dependent contrast of counterfactual representations, reconstructed at each rollout state; matched random and sign-reversed pushes test whether the effect is directional rather than generic perturbation.For robot tasks, edited scene copies are resynchronized with the acting copy after every step.Source: §D.1.1 · pp. 82–83
  • When the goal is lost in the representation, steering does not recover it: MAZE cheese reach is 0.001 unsteered and at most 0.002 under any push, while LOCATIONREACH never touches the ball first.The GM policies barely register the goal edits, so the constructed directions are much smaller than those for Non-GM policies.Source: §D.1.1 · p. 84
  • When goal information reaches the policy but is not used, corrective pushes also fail: PUSHTOMARKER reach is 0.000 at β = 1 and at most 0.018 at β = 4, while COIN stays between 0.016 and 0.027.The PUSHTOMARKER push weakens pursuit of the centre without moving the cube to the marker; in COIN, the goal edit changes the scene but supplies little orientation toward the coin.Source: §D.1.1 · pp. 85–86
  • Goal reach rises under state-local steering from 0.160 to 0.953 in ATTRIBUTE MAZE, 0.040 to 0.915 in ATTRIBUTETABLE, and 0.000 to 0.910 in SORTZONES at β = 1.WALKTOFLAG flag-first reach rises from 0.312 to 0.971; norm-matched random pushes stay near baseline, while reversed pushes reduce reach.Source: Figure 35 · p. 84, Table 14 · p. 85, §D.1.1 · p. 86
  • Steering also changes target selection, not merely activity: ATTRIBUTE MAZE conditional goal preference rises from 0.235 to 0.993 at β = 1, while larger doses can disrupt robot-task reach.At β = 2, robot reach falls to 0.278 in ATTRIBUTETABLE and 0.303 in SORTZONES.Source: §D.1.1 · p. 86
  • Successful steering can direct policies to the goal location even when the goal cue is removed or moved, so the authors characterize it as location redirection rather than recovered goal-cue pursuit.With the cue removed, the push reaches the original goal location in 0.913 of ATTRIBUTE MAZE states, 0.900 of ATTRIBUTETABLE scenes, and 0.970 of SORTZONES scenes.Source: §D.1.1 · pp. 86–87, Table 15 · p. 87

D.1.2 HOW MUCH CONTROL SURVIVES COMPRESSION ONTO A SHARED AXIS?

This section examines how much goal-directed control survives fixed steering along a compressed shared axis and whether rewiring frozen representations can recover it, specifying the interventions and evaluations used to test where recovery depends on surviving goal information. It also defines the state banks and episode protocol used to analyze policies and qualify behavior.

D.1.2 HOW MUCH CONTROL SURVIVES COMPRESSION ONTO A SHARED AXIS?

  • At β = 0.25, the sign-gated shared axis reaches 0.587 in ATTRIBUTE MAZE, 0.300 in ATTRIBUTETABLE, and 0.479 on WALKTOFLAG’s forced-view set; fixed SORTZONES pushes remain at 0.000.In ATTRIBUTE MAZE, 0.560 is the result for both the Fisher-projected axis and the fixed local direction.Source: §D.1.2 · pp. 88–89
  • Reversed sign-gated pushes reduce goal pursuit in Non-GM policies in ATTRIBUTE MAZE, ATTRIBUTETABLE, and SORTZONES, but not WALKTOFLAG, where fixed pushes leave reach near ceiling.The shared axis is scoped to the canonical UP-versus-RIGHT conflict class, not a universal goal-versus-proxy direction; large fixed pushes can also collapse rollouts into repeated movements.Source: §D.1.2 · pp. 89–90
  • The sign-gated axis works where one shared action contrast captures competition, while removing its sign largely cancels its effect across opposite conflict orientations.In ATTRIBUTE MAZE, 74% to 86% of the Fisher energy lies on the axis and unsigned steering gains 0.140 versus 0.427 for signed steering; in ATTRIBUTETABLE, the axis carries 57% and unsigned pushes also help, indicating a sign-independent component. The Fisher-projected push is weaker than the sign-gated push, especially in ATTRIBUTETABLE (0.108 versus 0.300).Source: §D.1.2 · pp. 89–90

D.2.1 DESIGN

  • Frozen-representation rewiring replaces downstream policy computation while retaining the trained representation, testing whether new wiring can recover goal-directed behavior when the native policy does not use goal structure successfully.The head rewire learns only a policy readout from hfc; the spatial rewire starts at hpre and relearns both the transformation to hfc and the policy head, testing whether goal-relevant spatial structure remains accessible after the learned late transformation.Source: §D.2.1 · pp. 90–91
  • Rewiring uses behavior cloning with teacher targets across all eight environments and oracle targets in MAZE and ATTRIBUTE MAZE, then evaluates intended-goal reach on held-out rollouts.Teacher targets come from competent Non-GM policies; oracle targets specify shortest-path actions toward the intended goal. The design separates reaching either candidate from selecting the goal conditional on reaching one.Source: §D.2.1 · pp. 91, 93, 94
  • The design compares rewires with native policies, Non-GM rewires, random trunks, goal-blind chance and parameter-matched random remixes, and defines recovery as at least 90% of a Non-GM reference.Successful rewiring shows that a frozen representation supports goal-directed control under the tested readout, supervision and state distribution; failure is less conclusive because downstream architecture, supervision, optimization or state distribution may limit recovery.Source: §D.2.1 · pp. 94–96

D.2.2 DOES NEW WIRING RECOVER THE GOAL?

  • From hpre, GM rewires recover the goal in COIN, ATTRIBUTE MAZE, SORTZONES and WALKTOFLAG, partly in PUSHTOMARKER and ATTRIBUTETABLE, and not in MAZE or LOCATIONREACH.From hfc, no GM rewire is recovered; only PUSHTOMARKER and WALKTOFLAG are partial. The patterns align with whether the goal survives in the representation.Source: §D.2.2 · p. 96, Table 18 · p. 97, Figure 38 · p. 98
  • DAgger makes the method viable in MAZE and PUSHTOMARKER, but only PUSHTOMARKER partially recovers: its hpre GM rewire reaches 0.81 of the Non-GM rewire, while MAZE still fails.In PUSHTOMARKER, hpre reach rises from 0.296 to 0.661 under DAgger, versus 0.311 for matched extra training; MAZE remains unrecovered despite the method becoming viable.Source: §D.2.2 · pp. 96, 99, Table 19 · p. 99

D.2.3 FROM WHICH LAYER CAN THE GOAL BE RECOVERED?

  • The goal survives the convolutional trunk but is usually lost in the learned hpre-to-hfc map, which compresses spatial features while retaining the cue the GM policy acts on: its proxy.The learned hfc is no better than a fixed random remix for GM agents in COIN and ATTRIBUTE MAZE, and only slightly better in LOCATIONREACH, PUSHTOMARKER and SORTZONES; it keeps clearly more in ATTRIBUTETABLE and WALKTOFLAG.Source: §D.2.3 · pp. 100–101

D.2.4 IS THE RECOVERY GOAL-DIRECTED?

  • Goal-removal tests support goal-directed recovery in PUSHTOMARKER, ATTRIBUTETABLE, SORTZONES and WALKTOFLAG, where removing or masking the cue reduces GM reach to about zero or below chance.In MAZE and LOCATIONREACH, GM rewires do not recover, while Non-GM rewires lose reach when goal cues are masked or removed. The COIN test is inconclusive because the coin often lies along the route to the level end.Source: §D.2.4 · p. 102
  • In ATTRIBUTE MAZE, the GM rewire follows the goal’s colour when its shape is removed, rather than reverting to the proxy.The red-gem test yields 0.521 goal reach for the GM rewire and 0.028 reach to the yellow-gem proxy; the cloning bank pairs red lines with yellow gems, confounding goal and colour cues.Source: §D.2.4 · p. 102

E EVALUATION BANKS

  • The evaluation banks are fixed before outcomes are computed, and each counted state includes a complete environment configuration and its rendered observation.Table 21 lists the banks and analyses that use them.Source: E EVALUATION BANKS · p. 104

E.1 COMMON TERMS

  • A bank is a fixed state list shared across policies, while conflict states place the intended goal and proxy at different locations.In a strict conflict bank, no action advances toward both targets, so the chosen action indicates which target the policy follows.Source: §E.1 · p. 104
  • World-edit banks support robot world-edit and Fisher analyses; probe banks vary goal and proxy independently, and cloning banks supply driver-policy rollouts for fitting downstream layers.In Procgen, the conflict bank serves as the world-edit bank; robot cloning banks are counted in frames.Source: §E.1 · p. 104
  • Axis fitting uses one bank partition and evaluates on the held-out portion, keeping each state and its mirror image on the same side of the split.A decision frame is taken mid-episode as the agent is about to commit to a candidate, rather than at the start.Source: §E.1 · p. 104
  • Qualification plays 500 episodes per checkpoint and distribution in every environment, using the same starting states across checkpoints within an environment.Robot conflict rollouts use the first 250 conflict episodes per arm task, while WALKTOFLAG uses the forced-view conflict bank.Source: §E.1 · p. 104

E.2 CONFLICT AND WORLD-EDIT BANKS

The appendices define matched banks for world edits, conflicts, probes, and rewiring, then set behavioral criteria for classifying goal-misgeneralized and control checkpoints.

E.2 CONFLICT AND WORLD-EDIT BANKS

  • World edits change only the goal or proxy in matched states, letting every policy be scored on identical pairs.Robot banks use task-specific sampler seeds; the five banks cover arm tasks and WALKTOFLAG.Source: §E.2 · pp. 104, 106

E.3 SPECIAL BANKS

  • Special banks add detours, diversified conflicts, off-path states, control layouts, and WALKTOFLAG views to test behavior beyond the basic conflict scenes.The ATTRIBUTE MAZE diversified bank spans every pair of movement directions, separating bearing and immediate action from the UP-versus-RIGHT conflict bank.Source: §E.3 · pp. 108–109

E.4 PROBE BANKS

  • Probe banks provide labelled variation in the concept being tested, while robot probe scenes vary goal and proxy independently.The banks include scripted or rollout-derived states and held-out partitions for probe evaluation.Source: §E.4 · pp. 109–110

E.5 CLONING BANKS

  • Rewiring uses behavior-cloning banks to fit fresh downstream layers, with the same bank serving matched GM and Non-GM rewires and each representation site.Procgen banks use driver rollouts; robot banks pool sampled-action rollouts from task-specific drivers.Source: §E.5 · pp. 110–111

F BEHAVIORAL QUALIFICATION OF GOAL MISGENERALIZATION

  • Behavioral qualification labels checkpoints from rollout outcomes before inspecting their representations, treating goal misgeneralization as graded rather than binary.Checkpoint labels are selected using explicit rules for sufficient proxy-following and control behavior.Source: F BEHAVIORAL QUALIFICATION OF GOAL… · p. 112

F.1 DEFINITIONS

  • GM checkpoints must show proxy-following while remaining capable; controls must also be capable, with the proxy uninformative during training.Checkpoints that are incapable or fall between the proxy-following thresholds enter neither cohort.Source: §F.1 · p. 112

F.2 ESTIMATES AND THRESHOLDS

The section defines behavioral thresholds and shows that goal pursuit varies continuously across checkpoints, while the selected robot cohorts are cleanly separated.

F.2 ESTIMATES AND THRESHOLDS

  • The proxy-following threshold is 0.25 and the control threshold is 0.02; capability thresholds are 0.40 for Procgen and 0.85 for robot tasks.A COIN check additionally requires every GM checkpoint to have dep < −0.05; visible-coin rendering lowers collection for all three, unlike control 7001.Source: §F.2 · pp. 113–114

F.3 GOAL PURSUIT IS GRADED

  • COIN checkpoints span Pr(goal | dec) = 0.402 to 0.788 across training, while robot GM checkpoints largely commit to the proxy (c Pr(goal | dec) ≤0.069).The authors caution that the robot results show cohort separation, not that robot training avoids Procgen-like intermediate values.Source: Figure 43 · p. 114, §F.3 · p. 114
  • Goal misgeneralization is graded: training does not distinguish the coincident goal and proxy, so intermediate goal preferences can occur among checkpoints.Only four of 63 scored greedy COIN checkpoints in the GM training regime meet the strict GM criterion and the additional dep check.Source: §F.3 · p. 114

F.4 THE COHORTS WE STUDY

  • The cohorts use three checkpoints per label, each meeting its definition; COIN GM checkpoints are selected by training step rather than being final run saves.Table 24 reports the worst checkpoint in each cohort, and the robot cohorts use final checkpoints.Source: Table 24 · p. 115, §F.4 · p. 115

G RELATED WORK IN DETAIL

The related work situates goal misgeneralization among behavioral proxy failures, distinguishes it from reward hacking, and contrasts this study’s internal diagnoses and post-training recovery with prior approaches.

Goal misgeneralization

  • The paper maps prior proxy categories to its three Procgen regimes: directional proxies to COIN, location proxies to MAZE, and observation ambiguity to ATTRIBUTE MAZE.It cautions that COIN does not distinguish a rightward rule from endpoint-directed behavior.Source: G RELATED WORK IN DETAIL · p. 117

Goals in reinforcement learning and reward hacking

  • Here, “goal” means a task-level outcome pursued by the trained network; unlike reward hacking, the reward stays fixed while the world changes.The authors distinguish this from goal-conditioned reinforcement learning, where the reward defines the goal.Source: G RELATED WORK IN DETAIL · p. 117

Goal misgeneralization and related failures in robot control

  • Prior robot work studied imitation-learning failures, but the authors report no prior demonstration of goal misgeneralization in robot policies trained by reinforcement learning.They induce it in five robot tasks under the same correct-reward and co-located-proxy conditions as in Procgen.Source: G RELATED WORK IN DETAIL · pp. 117–118

Interpretability of RL agents

  • Prior interpretability studies examine individual agents, while this paper applies matched internal assays across eight environments to distinguish different failures under one behavioral label.The authors note that their ATTRIBUTE MAZE swaps do not rule out a channel-specific shape detector as an explanation of cue effects.Source: G RELATED WORK IN DETAIL · p. 118

Representation versus use: probes, interventions and Fisher sensitivity

  • The paper tests whether representations are used, not merely decodable, by tracing policy sensitivity along world-edit displacements in activation-space Fisher geometry.It also uses state-local steering and behavior-cloning rewiring as post-training recovery tests, rather than mitigation during training.Source: G RELATED WORK IN DETAIL · pp. 118–119

H IMPLEMENTATION AND TRAINING DETAILS

The paper reuses established Procgen recipes, trains robot agents with platform baselines, modifies code to support matched edits, and documents AI assistance.

H IMPLEMENTATION AND TRAINING DETAILS

  • Procgen agents use the recipes that reported the failures, while robot agents use PPO baselines; within each environment, cohorts differ only in training distribution.COIN GM checkpoints are selected before run completion, and robot runs use a constant learning rate.Source: H IMPLEMENTATION AND TRAINING DETAILS · p. 120

H.1 MODIFICATIONS TO THE TRAINING CODE

  • Code changes enable state restoration and matched edits in Procgen, while the robot environments and trainers are implemented against ManiSkill3 and Isaac Lab APIs.The authors state that released code will reproduce the agents and analyses.Source: §H.1 · p. 122

AI USE STATEMENT

  • Generative AI assisted with prose, mathematics, literature search, analysis code, figures, and number audits; the authors reviewed and verified its outputs.The authors attribute the ideas and scientific claims to themselves.Source: AI USE STATEMENT · p. 122

Related papers

Loading 2610.19731v2…