PaperlayerView exact revision on arXiv

Source-linked AI summary

Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learning

S K Swaminathan, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan, Aritra Hazra

arXiv:2610.05087v1cs.LGcs.AIcs.RO

TL;DR

CRL’s policy receives raw goals rather than directly using the route geometry encoded by its critic. DCP conditions the actor on representation-space direction and distance, using visited waypoints during training and the final goal at deployment. It improves final success and time near goal over CRL on seven tasks each, while analyses identify waypoint selection and candidate availability as remaining constraints.

  • Problem

    CRL learns goal-reachability representations, but its policy remains conditioned on raw goals rather than directly using the geometry encoded by its critic.

    Source: §1 · p. 1

  • Method

    DCP conditions the actor on direction and distance in the critic’s representation, using scored visited states as training waypoints and the final goal directly at deployment.

    Source: §1 · p. 1, §3.3 · p. 4, §5 · p. 9

  • Results

    On seven tasks each, DCP improves final mean success and time near goal over CRL.

    Source: §5 · p. 9

  • Takeaways & Limitations

    DCP’s representation captures route structure and the actor uses its supplied direction, while learned candidate generation improves exploration in two controlled mazes.

    Source: §5 · p. 9

    Waypoint availability and ranking constrain exploration, and improving candidate availability and selection does not resolve AntSoccer.

    Source: §4.6 · p. 9, §5 · p. 10

Abstract

from arXiv · show

Contrastive Reinforcement Learning (CRL) learns representations that estimate goal reachability, yet its policy remains conditioned on raw goals and therefore does not directly exploit the geometry encoded by its critic. We introduce Direction-Conditioned Policies (DCP), a method built around a small modification to CRL: DCP selects previously visited states as waypoints during online training and conditions the policy on their direction and distance in representation space. At deployment, DCP applies the same interface directly to the final goal, requiring neither waypoint selection nor planning. Across nine navigation and manipulation tasks, DCP attains higher final success rates than CRL on seven tasks and spends more time near the goal on seven. Controlled maze experiments further show that DCP captures shortest-path geometry more accurately and that the supplied direction causally influences the actor's behavior. We identify waypoint coverage and ranking as limits to exploration, and show that learned candidate generation improves goal reaching in two controlled mazes.

1 INTRODUCTION

CRL learns from achieved states, but its policy still receives raw goals, whose coordinate differences can misrepresent routes around obstacles. DCP conditions the actor on direction and distance in the critic’s representation, using visited waypoints during training and the final goal at deployment.

1 INTRODUCTION

  • Raw goal coordinates can point through a maze wall, while the critic’s learned representation can capture route structure.Distant goals may have unreliable learned directions early in training, motivating visited states as waypoint candidates.Source: §1 · p. 1
  • DCP encodes the current achieved goal and target, then gives the actor their normalized representation-space displacement and its magnitude.The physical state is also supplied to the actor.Source: §1 · p. 1
  • At deployment, DCP replaces the waypoint with the final goal, so the same policy interface needs no waypoint pool or scorer.Figure 1 contrasts raw-goal CRL, waypoint-conditioned SSGC, and direction-and-distance DCP inputs.Source: §1 · p. 2, Figure 1 · p. 2
  • DCP improves final mean success and time near goal over CRL on seven of nine tasks each.The comparison is reported across five seeds.Source: §1 · p. 2
  • Controlled experiments examine route geometry, the actor’s use of direction, and waypoint selection; learned candidate generation improves goal reaching in two controlled mazes.Improving selected waypoints alone does not resolve the more disconnected, dynamic AntSoccer task.Source: §1 · p. 2

2 RELATED WORK

Prior work develops contrastive representations for control, directional policies and goal representations, and relative subgoals. This paper studies how learned goal relations can be reused by an online contrastive actor–critic.

Contrastive representations for control

  • DCP reuses critic goal-encoder features in the actor while retaining the baseline network depth and single-action policy.Related CRL work includes accelerated simulation, depth scaling, action-sequence conditioning, and directed exploration before a first successful trial.Source: §2 · p. 2

Direction-conditioned skills and goal representations

  • Prior methods learn directional policies or representations through offline data, online unsupervised exploration, or compositional plan representations.Examples include HILP, METRA, and plan arithmetic.Source: §2 · p. 2
  • Relative subgoals appear in hierarchical policies and latent-skill methods, while learned temporal distances and quasimetrics provide underlying geometry.DCP brings learned goal relations to an online contrastive actor–critic and learns its representation from policy-collected experience.Source: §2 · p. 3

Interpolation and waypoint substitution

  • Prior work studies when an on-path waypoint and final goal induce the same action; this paper tests substitution and scorer reliability.The related work also notes this property for flat policies bootstrapped from subgoal-conditioned ones.Source: §2 · p. 3

3 METHODOLOGY

DCP retains contrastive goal-conditioned learning while changing the actor’s goal input to direction and distance in the critic’s representation. Collection uses scored visited states, learning uses achieved future goals, and deployment uses the requested final goal.

3.1 CONTRASTIVE GOAL-CONDITIONED LEARNING

  • CRL learns state–action and goal encoders, with negative Euclidean-distance energy and backward InfoNCE distinguishing matching state–action pairs for sampled future goals.The method retains baseline log-sum-exp regularization, replay sampling, and the entropy-regularized actor objective.Source: §3.1 · p. 3

3.2 A RELATIVE GOAL INPUT FOR THE ACTOR

  • DCP replaces the raw-goal actor input with the normalized difference between target and achieved-goal embeddings plus their distance.The full physical state remains an input because actions also depend on velocities and body configuration; direction and radius jointly recover displacement up to numerical stabilization.Source: §3.2 · p. 3

3.3 COLLECTION, LEARNING, AND DEPLOYMENT

  • During collection, DCP samples visited-state candidates and scores them against the final goal; replayed trajectories supply achieved future goals for actor learning.The selected target is held until refresh while its direction is recomputed each action; at deployment the final goal is used directly, removing the pool and scorer.Source: §3.3 · pp. 3–4

4 EXPERIMENTS

Across nine tasks, DCP improves final success and time near goal over CRL on seven tasks each. Controlled experiments examine its learned route geometry, the actor’s use of direction, and how waypoint selection and candidate generation affect performance.

4.1 DOES DCP IMPROVE ONLINE GOAL REACHING?

  • DCP improves final mean success and time near goal over CRL on seven tasks each, and final mean success over SSGC on seven tasks.Its largest absolute success gains over CRL are +51.0 percentage points on PusherEasy, +34.9 on PusherHard Far, and +18.9 on PusherHard; AntSoccer is a failure case on both measures.Source: §4.1 · pp. 4–5, Table 2 · p. 5

4.2 WHAT DO SCORING AND DIRECTION CONDITIONING CONTRIBUTE?

  • Direction conditioning without waypoint selection has the largest mean success in both settings, while adding scoring changes mean success by −0.105 in the open room and −0.032 in the U-maze.The U-maze difference has a paired 95% interval of [−0.125, 0.061], so waypoint selection can slow learning.Source: §4.2 · pp. 5–6, Table 3 · p. 6

4.3 DOES THE REPRESENTATION CAPTURE OBSTACLE-RESPECTING ROUTES?

  • DCP’s goal-embedding distances correlate more strongly with U-maze graph distances than CRL’s, and their negative spatial gradients point to a graph-optimal first move on 70.9% versus 64.7% of route-conflict pairs.DCP’s U-maze success over training is similar to CRL’s (0.847 versus 0.846), so better distance ordering alone does not explain control performance.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4 DOES THE ACTOR USE THE SUPPLIED DIRECTION?

  • In the U-maze, reversing the learned direction reduces success from 45.8% to 0.2%; in puck pushing, reversing direction reduces DCP success to 2.9%.DCP succeeds in 35.9% of matched puck-pushing cases versus 27.9% for CRL, and makes correct-side contact in 55.1% versus 43.7%.Source: Figure 5 · p. 8, §4.4 · p. 8

4.5 WHEN DOES A WAYPOINT PRESERVE THE FINAL-GOAL ACTION?

  • For point-mass policies, an exact on-path midpoint target yields mean action cosine 0.789 versus −0.072 for an equally distant off-path target.Positive alignment with a graph-optimal first move occurs in 95.2% versus 41.9% of cases, and the on-path target performs better across all seeds.Source: §4.5 · p. 8

4.6 HOW WELL DOES THE SCORER SELECT USEFUL WAYPOINTS?

  • An exact forward on-path candidate is available in 77.7% of draws, but the scorer selects one in only 12.9–18.8%, despite improving over uniform selection at 8.3%.Increasing candidate availability from 77.9% to 94.3% raises useful selection by only 2.4–3.9 percentage points; when the pool has no useful state, more draws cannot help.Source: §4.6 · pp. 8–9

4.7 CAN A LEARNED CANDIDATE SOURCE IMPROVE EXPLORATION?

  • A learned candidate generator increases mean success over training in both controlled mazes, beating the pool in four of five paired seeds in the two-route maze and all five in the rare-gateway maze.Projecting proposals to stored states improves the two-route mean but removes the rare-gateway benefit, where proposals can land on the wrong side of the passage.Source: Table 5 · p. 9, §4.7 · p. 9

4.8 DOES SEEDING THE POOL WITH SUCCESSFUL STATES HELP?

  • On PusherHard, reserving 128 of 512 slots for successful-path states raises mean success from 32.39% to 38.17%, but adding those states while preserving all online slots gives no consistent benefit.On AntSoccer, successful-path states are selected more often and improve waypoint-to-goal direction agreement, but ball motion and success do not improve.Source: §4.8 · p. 9

5 CONCLUSION

DCP uses the critic’s learned route geometry to condition the actor, outperforming CRL on two final benchmark measures across most tasks. Analyses expose waypoint-selection challenges, an unresolved task, and the study’s simulated state-observation scope.

5 CONCLUSION

  • Across nine tasks, DCP improves final mean success and time near goal over CRL on seven tasks each.The analyses also report that the representation captures route structure and the actor uses supplied direction in navigation and manipulation.Source: §5 · p. 9
  • Waypoint selection can slow learning when direct goal conditioning already works, and better candidates help only when the policy can act on them.Learned generation improves exploration in two controlled mazes, while adding successful states to the pool leaves AntSoccer unresolved.Source: §5 · pp. 9–10
  • The experiments use simulated state observations; future work could extend DCP to visual observations and combine it with deeper or action-chunked CRL.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points to implementation materials and appendices for the method, settings, and experimental reporting.

REPRODUCIBILITY STATEMENT

  • The implementation and reproduction scripts are available through the linked DCP supplement.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix A specifies objectives, policy inputs, and collection, while Appendix B gives task definitions, budgets, and experimental settings.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Final task comparisons use five seeds; Figure 3 and Appendix G provide learning curves and checkpoint values.Additional experiments state their seed counts separately.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement confines the study to simulated state observations and reports no human-subject data; physical use would require further validation.

ETHICS STATEMENT

AI USE STATEMENT

The supplementary material describes AI assistance, the default training procedure, matched experimental settings, and parameter and compute costs.

AI USE STATEMENT

  • Generative AI assisted with code, result analysis, literature retrieval, and manuscript drafting and revision; the authors retain responsibility for the final work.Source: AI USE STATEMENT · p. 10

A.1 DEFAULT ALGORITHM

  • The default algorithm selects a scored visited-state waypoint during collection, relabels replay with future achieved goals, and uses the final goal directly at deployment.Its actor input concatenates full state, a 64-dimensional direction, and one radius.Source: §A.1 · p. 13

A.2 TRAINING SETTINGS

  • DCP, CRL, and SSGC use the same within-task simulation settings and backward InfoNCE configuration, with comparisons based on matched runs.Source: §A.2 · pp. 13–14, Table 6 · p. 14

A.3 PARAMETER COUNTS

  • DCP replaces raw goal coordinates with 64 direction components and one distance, adding input-layer weights while leaving remaining actor layers and critic encoders unchanged.Across benchmark tasks, the actor parameter increase is 11–21%, and the complete actor–critic increase is 4.15–6.40%.Source: §A.3 · p. 14
  • DCP runs at 95.3% of CRL’s median throughput across tasks, with ratios ranging from 92.4% to 97.6%.The waypoint pool and scoring are removed at deployment, though goal-encoder evaluation remains required.Source: §A.3 · p. 14

B TASKS AND EVALUATION PROTOCOLS

The paper specifies benchmark and controlled-maze protocols, then examines waypoint selection, representation geometry, candidate generation, and demonstration-state libraries. In a capacity-limited pool, successful-path seeding beats matched pre-motion seeding, but adding those states to the full pool does not help.

B TASKS AND EVALUATION PROTOCOLS

  • The nine benchmark tasks match interaction budgets across methods, define success by task-specific goal coordinates and radii, and report success, time near goal, and training averages under stated seed protocols.The controlled studies use separately specified environments and protocols, with uncertainty calculated across independent training seeds.Source: B TASKS AND EVALUATION PROTOCOLS · pp. 14–15, Table 8 · p. 15, Table 9 · p. 16

C.1 CONTROLLED ENVIRONMENTS

  • Controlled environments include point-mass mazes, a differential-drive U-maze, and puck pushing, with matched within-layout settings and tests designed to probe graph geometry and direction-conditioned behavior.The U-maze graph compares learned distances with route lengths and graph-optimal moves; the puck test varies approach side while holding puck and target fixed within groups.Source: §C.1 · pp. 15–17

C.2 GRAPH REFERENCE AND INTERPRETATION

  • The U-maze graph uses weighted cardinal and collision-free diagonal connections, and its 100,128 directed position pairs provide route-distance references for global-distance and local-direction probes.The open room is a setting where graph and Euclidean distances largely agree; scoring-rule and actor-input comparisons are conducted within each layout.Source: §C.2 · p. 17

C.3 COMPARING WAYPOINT SELECTION RULES

  • Across five trained encoders, cosine ranking selects forward on-path candidates in 24.7–27.8% of draws and latent-slack ranking in 28.6–33.0%, versus 12.9–18.8% for default inner-product ranking.Uniform selection achieves 8.3%; the test fixes encoders and candidate sets, so online-learning effects remain untested.Source: §C.3 · p. 17

C.4 A GEOMETRIC STATEMENT ABOUT THE DIRECTION INPUT

  • Proposition 1 relates latent triangle slack to direction agreement, while the learned embedding distance is only a pseudometric and needs additional assumptions to relate it to optimal-control cost.The generator trains on replay-sequence midpoints and proposes waypoint coordinates; projection to the nearest pool state can move proposals across obstacles.Source: §C.4 · pp. 17–18, §D.1 · p. 18

D.2 DEMONSTRATION-STATE LIBRARIES

  • Successful-path seeding exceeds matched pre-motion seeding in all three seeds when replacing capacity-limited pool states, while augmentation of the full online pool provides no corresponding benefit.The comparison depends on both library content and which online candidates the library replaces.Source: §D.2 · p. 19
  • For AntSoccer, successful-path seeding raises mean waypoint/final-goal direction cosine from 0.596 to 0.655 and lowers latent slack from 1.096 to 1.000, without improving ball motion or success.The authors conclude that better candidate availability and selection alone are insufficient for this task.Source: §D.2 · p. 19

E.1 LEARNING EARLY IN TRAINING AND MAINTAINING SUCCESS

In three-seed analyses, DCP generally crosses early success thresholds sooner and maintains success after crossing them. AntSoccer combines sparse ball movement with little change in distance ordering; a frozen-CRL transfer succeeds in one AntMaze Big run, with generality still untested.

E.1 LEARNING EARLY IN TRAINING AND MAINTAINING SUCCESS

  • DCP crosses 2% success earlier than CRL on seven tasks and ties on two; its first-25M-step success is higher on seven, tied on AntPush, and lower on AntSoccer.At 5% and 10% thresholds, DCP crosses earlier on six tasks, ties on one, and later on two.Source: §E.1 · p. 19
  • After first crossing a threshold, the median fraction of the next ten evaluations below 5% and 10% is 0% for DCP, versus 10% and 20% for CRL.On AntPush, DCP mean success falls from 44.8% at 75M steps to 42.7% at 100M steps.Source: §E.1 · p. 19

E.2 ANTSOCCER REPRESENTATION AND ACTIVITY

  • AntSoccer shows little representation change and sparse object motion: mean Spearman-correlation change is −0.050 at 25M and −0.021 at the final checkpoint, while only 3.8% of retained transitions move the ball by more than 10−3.These signals identify AntSoccer as unusual in completed runs; predicting individual-run outcomes requires further testing.Source: §E.2 · p. 19

E.3 FROZEN CRL REPRESENTATIONS

  • With both CRL encoders frozen, a new DCP actor reaches 23.0% final success on AntMaze Big in one seed; fresh DCP reaches 25.4% with a smaller total interaction budget.The unequal pretraining budgets prevent a sample-efficiency comparison, and transfer generality across seeds and tasks remains untested.Source: §E.3 · pp. 19–20, Table 11 · p. 19

F ADDITIONAL VISUALIZATIONS

The additional visualizations compare learned geometry, waypoint scoring, and candidate generation. They show consistent global-distance ordering gains across seeds and illustrate how projection’s effect depends on maze layout.

Variation across training seeds

  • DCP has higher graph-distance correlation than CRL in every seed for both all state pairs and the longest-distance third, while local-descent results vary more across seeds.Figure 7 compares paired seed measurements after 10M steps.Source: F ADDITIONAL VISUALIZATIONS · p. 21, Figure 7 · p. 21

Distance fields in AntMaze Big

  • Distance-field contours and trajectories illustrate corridor-sensitive learned distances; one AntMaze Big example has both methods reach the goal, while another shows DCP reaching it and CRL stalling.Colors are scaled separately within panels, so compare spatial structure rather than absolute color values.Source: F ADDITIONAL VISUALIZATIONS · p. 21, Figure 8 · p. 21, Figure 9 · p. 22

Comparing waypoint scoring rules; Generating candidates beyond the visited-state pool

  • Direction-agreement and latent-slack ranking select available forward on-path candidates more often than default inner-product scoring, while generated candidates improve mean training success in both layouts and projection helps only in the two-route maze.Projection reduces success in the rare-gateway maze, where it can remove proposals beyond a poorly explored gateway; the generator comparison averages 20 checkpoints per run.Source: F ADDITIONAL VISUALIZATIONS · pp. 22–23, Figure 11 · p. 23, Figure 12 · p. 23

G COMPLETE BENCHMARK CHECKPOINTS

The checkpoint tables report success at least once per episode and time spent near the goal across five seeds. Their examples distinguish occasional arrival from sustained proximity and show task-specific performance patterns.

Success checkpoints

  • At PusherHard, DCP reaches 33.7% success at 25M steps versus CRL’s 0.9%, and 47.7% at 100M versus CRL’s 28.8%.The table reports whether the goal region is reached at least once per episode; bold means mark the largest mean, not statistical significance.Source: G COMPLETE BENCHMARK CHECKPOINTS · p. 24, Table 12 · p. 24
  • DCP mean AntPush success falls from 44.8% at 75M steps to 42.7% at 100M, while CRL has the higher mean at every reported AntSoccer checkpoint.The checkpoint tables report five-seed means with 95% confidence-interval half-widths.Source: G COMPLETE BENCHMARK CHECKPOINTS · p. 24, Table 12 · p. 24

Time-near-goal checkpoints

  • At final PusherEasy evaluation, DCP’s 55.4% success accompanies 297.8 steps near goal, versus CRL’s 4.4% and 26.2 steps.On Humanoid U-Maze, DCP reaches the goal in 8.4% of episodes but averages 3.6 steps near it, indicating brief arrivals.Source: G COMPLETE BENCHMARK CHECKPOINTS · p. 24, Table 13 · p. 25
Loading 2610.05087v1…