PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperspherical Contrastive Representations Unlock Planning and Exploration

Jonghae Park, Sungil Son, Jonghun Shin, Junseo Lee, H. Jin Kim, Benjamin Eysenbach

arXiv:2610.20409v1cs.LG

TL;DR

Long-horizon goal-reaching requires choosing a suitable amount of planning, but many hierarchical approaches use fixed waypoint counts or horizons. C-Slerp uses hyperspherical contrastive representations to infer waypoints in closed form and adapt their number; it solves long-horizon tasks, including settings where baselines achieve no success.

  • Problem

    Many hierarchical approaches use fixed waypoint counts or fixed subgoal horizons rather than selecting how many stages to plan for each state-goal problem.

    Source: §1 · pp. 1–2

  • Method

    C-Slerp composes hyperspherical contrastive representations to adapt waypoint counts and infer intermediate waypoints through closed-form spherical interpolation, without a separate high-level network or graph search.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    C-Slerp solves long-horizon navigation and manipulation tasks, including settings where baselines achieve no success; fixed waypoint counts substantially degrade performance.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results suggest that adapting task decomposition to the goal-reaching problem contributes to the benefits of hierarchy in goal-conditioned reinforcement learning.

    Source: §6 · p. 10

    C-Slerp requires extensive environment interaction, and the experiments use state observations in simulation, leaving visual learning and real-robot deployment untested.

    Source: §6 · p. 10

Abstract

from arXiv · show

We develop a self-supervised reinforcement learning algorithm that learns to build towers of up to eight cubes from scratch, without prior data, rewards, demonstrations, or pretrained models. Our method, Contrastive Planning via Spherical Linear Interpolation (C-Slerp), uses contrastive representations as reusable building blocks and repeatedly composes them to infer a sequence of intermediate waypoints. Under our hyperspherical parameterization, the learned representations allow intermediate waypoints to be inferred in closed form through spherical linear interpolation, without training a separate high-level policy or performing graph search. Importantly, the representation geometry also tells us how to adjust the number of waypoints based on the difficulty of each goal-reaching problem. Experiments show that our method succeeds on long-horizon navigation and robotic manipulation tasks, including settings where strong baselines fail to achieve a single success. Ablations further show that using a fixed number of waypoints substantially degrades performance, highlighting the importance of adapting the plan to task difficulty.

1 INTRODUCTION

Long-horizon goals need a suitable number of planning stages: too many can complicate simple tasks, while too few can leave difficult tasks unsolved. C-Slerp composes contrastive representations to select stages and infer waypoints, with results indicating that adaptive planning helps on challenging goal-reaching tasks.

1 INTRODUCTION

  • Planning stages should match task difficulty: too many can complicate a single-cube task, while too few can leave tower building difficult.The section contrasts placing one cube with building an eight-cube tower.Source: §1 · p. 1
  • Fixed waypoint counts or fixed-horizon subgoals are common in prior hierarchical approaches, rather than explicitly choosing a plan length for each problem.Waypoints divide long-horizon tasks into shorter subproblems.Source: §1 · p. 1
  • C-Slerp composes contrastive representations to score different plan lengths, select an appropriate number of stages, and infer the corresponding waypoints.Its spherical geometry supports closed-form waypoint inference without a separate high-level policy or graph search.Source: §1 · p. 2
  • C-Slerp solves long-horizon navigation and manipulation tasks, including settings where strong exploration methods achieve no success in one billion environment steps.Ablations find that fixing the number of planning stages substantially degrades performance.Source: §1 · p. 2

2 RELATED WORK

Prior hierarchical methods often fix the number or temporal horizon of subgoals, while graph-based planning can use variable-length waypoint sequences at added computational and state-coverage costs. C-Slerp jointly learns contrastive representations and a policy online, then uses the representations to adapt waypoint counts and infer waypoints in closed form.

2 RELATED WORK

  • Many hierarchical methods fix waypoint counts or subgoal horizons; graph methods allow variable waypoint counts but require graph construction and depend on graph-state coverage.The section frames C-Slerp as adapting waypoint count without a planning graph.Source: §2 · pp. 2–3
  • Prior work links hierarchy to exploration, reuse of repeated subproblem solutions, and clearer policy-learning signals, while arguing benefits are not simply due to policy expressivity.SAW is described as bringing hierarchy's benefits to flat policies through bootstrapping, without a separate subgoal generator.Source: §2 · p. 2
  • Contrastive reinforcement learning learns goal-reaching representations and policies self-supervised, but whether exploration and network scaling suffice for much longer horizons remains unclear.Prior closed-form waypoint distributions use offline data and require a predefined waypoint count.Source: §2 · pp. 2–3
  • C-Slerp jointly learns contrastive representations and a goal-conditioned policy online, adapting waypoint counts and inferring waypoint representations without a separate high-level policy or graph search.This design lets the paper examine how adaptive planning contributes to hierarchy's benefits on long-horizon tasks.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries establish an online, self-supervised goal-conditioned reinforcement-learning setting and describe contrastive reinforcement learning as a way to train a goal-reaching critic and policy from future-state data.

3 PRELIMINARIES

  • The paper focuses on fully online RL: the replay buffer updates during policy training, and the critic uses a self-supervised objective without external rewards.Future goals are sampled from replay-buffer states using temporal offsets drawn from GEOM(1 − γ).Source: §3 · p. 3
  • In goal-conditioned RL, a policy selects actions based on the current state and a sampled desired goal, with goal-reaching reward defined through the probability of transitioning to that goal.The corresponding Q-function is related to goal-reaching under a discounted state occupancy measure.Source: §3 · p. 3
  • Contrastive RL trains a goal-conditioned critic from batches whose goals are future states sampled at geometric temporal offsets, then trains the policy to maximize that critic.The critic's action ranking corresponds to discounted goal-reaching probability under the occupancy measure.Source: §3 · p. 3

4 HYPERSPHERICAL CONTRASTIVE REPRESENTATIONS FOR PLANNING

This section develops C-Slerp as a planning method that learns hyperspherical contrastive representations and uses them to infer waypoint representations and adapt the waypoint count. Its closed-form interpolation and selection rules are combined with online representation and policy learning.

4.1 PROBLEM FORMULATION

  • The planning problem is to infer intermediate waypoints between a state and goal, then select a suitable waypoint count for each state-goal pair.The setup collects data online with a hierarchical policy that periodically supplies waypoint representations to a low-level policy.Source: §4 · p. 4, §4.1 · p. 4

4.2 HYPERSPHERICAL CONTRASTIVE REPRESENTATIONS

  • The method represents states and state-action pairs on a unit hypersphere and uses a learnable rotation to model asymmetric temporal relationships while preserving the geometry.The rotation allows one temporal score to be applied repeatedly to consecutive waypoint pairs.Source: §4.2 · p. 4

4.3 CLOSED-FORM WAYPOINT INFERENCE

  • Closed-form spherical interpolation infers waypoint distributions from learned representations, accounting for temporal transitions through the rotation and inverse mappings without an additional neural network.For R = I, waypoint means lie evenly along the geodesic between the state and goal representations.Source: §4.3 · p. 5, Figure 2 · p. 6

4.4 HOW MANY WAYPOINTS ARE NEEDED FOR EACH STATE-GOAL PAIR?

  • The method selects waypoint count using the alignment of the goal representation with the direction predicted after K + 1 geometric transitions.It models this alignment probabilistically to infer K for each state-goal pair; the policy pursues the furthest candidate waypoint estimated reachable by its alignment threshold.Source: §4.4 · p. 6

4.5 LEARNING OBJECTIVES AND ALGORITHM

  • Online training alternates data collection with representation and policy updates, and the planner periodically selects a waypoint subgoal for the policy.Representation learning uses backward and symmetric InfoNCE objectives, while the policy maximizes its goal-conditioned critic.Source: §4.5 · pp. 6–7
  • C-Slerp achieves the highest success rates on most BuilderBench and JaxGCRL tasks, and is the only method to succeed on all gripper cube-stacking tasks.Figure 3 shows success-rate learning curves for manipulation and locomotion benchmarks.Source: Figure 3 · p. 7

5 EXPERIMENTS

C-Slerp achieves the highest success rates on most of twelve navigation and manipulation tasks, with its advantage growing on longer-horizon tasks. Ablations show that both waypoint count and waypoint choice need to adapt to the task and current state.

5 EXPERIMENTS

  • The evaluation spans six JaxGCRL locomotion tasks and six BuilderBench manipulation tasks, comparing C-Slerp with five goal-conditioned reinforcement-learning baselines.BuilderBench tasks vary in cube count and whether a gripper is used; both affect task horizon. Baselines include contrastive, hierarchical, graph-planning, and SAC + HER methods.Source: §5 · p. 7

5.1 EXPERIMENTAL RESULTS

  • C-Slerp achieves the highest success rates on most tasks and is the only method to succeed on every gripper cube-stacking task, where most baselines achieve no success.The methods perform comparably on shorter-horizon cube-stacking tasks; the gap widens as cube count increases or a gripper is used.Source: §5.1 · p. 8
  • As navigation horizons increase, C-Slerp achieves consistently higher success rates than several baselines that perform comparably on shorter-horizon tasks.The authors suggest that waypoint-based decomposition facilitates exploration by making challenging tasks into easier subproblems.Source: §5.1 · p. 8
  • Adaptive waypoint-count selection achieves the highest success rates on both the 8-tower task and Ant U5 Maze; fixed counts make little progress on 8-tower and do not consistently improve with more waypoints.The appropriate count can change within an episode: too few waypoints can leave subproblems difficult, while too many can introduce unnecessary stages.Source: §5.1 · p. 8
  • Adaptive waypoint selection achieves the best performance on both 8-tower and Ant U5 Maze, while the best fixed waypoint index differs between environments.Earlier waypoints can be conservative and slow progress; later ones can enable more progress but be harder for the current policy to reach.Source: §5.1 · p. 8
  • With a single fixed, distant goal in Ant Big Maze and Ant U4 Maze, C-Slerp achieves higher success rates than all baselines; all except CRL and L3P achieve zero success.This setting evaluates exploration with the goal fixed throughout training and evaluation.Source: §5.1 · p. 9
  • In the 5-tower gripper rollout, the selected waypoint is not always the first inferred waypoint, and fewer waypoints appear as the tower nears completion.At replanning steps, C-Slerp updates the waypoint count, infers waypoint representations, and selects a subgoal for the policy.Source: §5.1 · p. 9

6 CONCLUSION

C-Slerp combines adaptive waypoint planning with closed-form spherical interpolation to solve long-horizon navigation and manipulation tasks. Its experiments remain limited to state observations in simulation, and training requires extensive environment interaction.

6 CONCLUSION

  • C-Slerp solves long-horizon navigation and manipulation tasks, including settings where no baseline succeeds, and ablations favor adaptive waypoint count and selection over fixed choices.The conclusion says task decomposition adapted to the goal-reaching problem contributes to hierarchy's benefits in goal-conditioned reinforcement learning.Source: §6 · p. 10
  • Training requires extensive environment interaction, and the experiments use state observations in simulation rather than testing visual observations or real-robot deployment.The conclusion identifies data efficiency and pixel-based observations as future-work directions.Source: §6 · p. 10
  • Spherical linear interpolation infers waypoints in closed form without a separate high-level network or planning graph.The conclusion presents this as evidence that adaptive planning need not require a complex architecture.Source: §6 · pp. 9–10

A THEORETICAL RESULTS

The theoretical analysis relies on a Bayes-optimal form for the learned contrastive critic and assumes its representation marginal is uniform on the unit hypersphere. It motivates uniformity through the entropy term in the objective.

A THEORETICAL RESULTS

  • The analysis assumes the learned symmetric-InfoNCE critic attains its Bayes-optimal form; including the positive pair in the denominator changes the loss by at most 2 log(1 + e^2/(B − 1)), vanishing as B grows.The limiting result is transferred from the cited formulation, which excludes the positive pair.Source: §A.1 · p. 14
  • The assumed marginal distribution of representations is uniform on the unit hypersphere.The objective's entropy interpretation motivates this assumption: maximizing entropy on the sphere yields the uniform distribution.Source: §A.1 · p. 14
  • The uniformity justification decomposes the objective into alignment and uniformity terms, with the latter interpreted through a von Mises–Fisher kernel density estimate.The derivation approximates the log-denominator after assuming positive pairs are sufficiently aligned.Source: §A.1 · p. 14

A.2 DERIVATION OF THE REPRESENTATION-SPACE OBJECTIVE

The appendix derives a representation-space waypoint objective from a posterior KL objective, then gives an approximate closed-form solution. Under a small-variance approximation, waypoint mean directions lie at equal angular intervals along the shortest spherical geodesic.

A.2 DERIVATION OF THE REPRESENTATION-SPACE OBJECTIVE

  • The derivation starts from a posterior KL objective for K intermediate waypoints and factorizes the waypoint distribution into one conditional distribution per waypoint.It expresses the objective in representation space using densities induced by the mapping ψ.Source: §A.2 · p. 15
  • Under the stated finite-entropy condition, choosing each waypoint conditional to match the data-collection policy's transition distribution makes the conditional KL equality achievable.This allows minimizing KL over representations to lead to minimizing the original waypoint KL.Source: §A.2 · p. 15

A.3 DERIVATION OF THE CLOSED-FORM SOLUTION

  • The closed-form approximation optimizes waypoint mean directions and variances separately, using small-variance assumptions and an entropy weight to control waypoint stochasticity.The tangent-plane approximation treats each waypoint representation as concentrated around its mean direction.Source: §A.3 · p. 16
  • The optimal waypoint directions are equally spaced along the shortest geodesic between the transformed start and goal representations, with angular interval θ/(K + 1).The derivation excludes θ = π because the shortest arc is not unique; the θ = 0 case follows by continuity.Source: §A.3 · p. 16
  • The approximate variance solution is a unique maximizer for d ≥ 2 and α > 0 when the small-variance condition α(d − 1)/(2 cos ∆) ≪ 1 holds.The objective is strictly concave in each positive waypoint variance under these conditions.Source: §A.3 · p. 17

A.4 DERIVATION OF THE DISTRIBUTION OVER THE NUMBER OF WAYPOINTS

The appendix estimates how well a sequence of geometric transitions aligns with the goal representation, then uses that alignment to form a distribution over waypoint counts. The resulting score compares candidate counts through their predicted representation directions.

A.4 DERIVATION OF THE DISTRIBUTION OVER THE NUMBER OF WAYPOINTS

  • With K intermediate waypoints, the trajectory has K + 1 segments, each using an independently sampled geometric time interval.For K = 0, the resulting goal distribution is the single geometric-transition distribution.Source: §A.4 · p. 18
  • Under the stated uniformity and high-dimensional assumptions, the expected representation after K + 1 geometric transitions is approximated by recursively composing the one-step prediction.The predicted direction is normalized and aligned with the goal representation to score K.Source: §A.4 · pp. 18–19
  • A von Mises–Fisher likelihood centered at the predicted direction converts goal alignment into a posterior distribution over K, combined with a prior over waypoint counts.With a uniform prior over the allowed counts, the posterior simplifies accordingly.Source: §A.4 · p. 19

B.1 ABLATION STUDY

The ablations examine waypoint-selection thresholds, representation design, and whether extending CRL’s temporal horizon can match adaptive waypoint planning. They show task-dependent sensitivity to some settings and no comparable benefit from simply increasing CRL’s discount factor.

B.1 ABLATION STUDY

  • C-Slerp is relatively robust to ϵ ∈ {0.2, 0.3, 0.4} except on 8-tower, where ϵ = 0.4 performs best.By contrast, ϵ = 0.2 and 0.3 perform better on Ant U5 Maze; the default is ϵ = 0.4.Source: §B.1 · p. 20
  • Learning R supports stable performance: R = I prevents successful learning, while randomly initialized fixed R permits learning but is unstable.The results suggest that asymmetry alone is insufficient and that learning R helps capture temporal relationships between states.Source: §B.1 · p. 20
  • Deeper networks generally achieve higher success; on 8-tower, depths 16 and 32 perform comparably and outperform shallower networks.On Ant U5 Maze, performance improves as depth increases.Source: §B.1 · p. 21
  • On 8-tower, only tested representation dimensions d ≥ 256 achieve nonzero success, and further increases provide no consistent benefit.The authors hypothesize that the encoder ψ already supplies sufficient nonlinear expressivity for the linear operator R.Source: §B.1 · p. 21

B.2 CRL WITH LARGER DISCOUNT FACTORS

  • Increasing CRL’s γ to 0.995 or 0.999 did not close its low-success gap with C-Slerp, which uses γ = 0.99.The authors suggest that weighting distant future states more heavily may make temporally close and distant goals harder to distinguish.Source: §B.2 · p. 22

B.3 VISUALIZATIONS

The visualizations show how C-Slerp’s waypoint count changes as the agent approaches a goal and how inferred latent waypoints relate to environment rollouts and maze positions. The accompanying experimental details describe the evaluated navigation and manipulation settings and their success criteria.

B.3 VISUALIZATIONS

  • The mean number of transition segments generally decreases as the agent approaches the goal, suggesting that the representation captures remaining temporal distance.In Humanoid U Maze, the count increases again after the humanoid reaches the goal, falls, and cannot recover.Source: §B.3 · p. 23
  • Across environments, rollout visualizations show C-Slerp adjusting waypoint counts and selecting latent subgoals as the agent progresses toward its goal.Figures 15 and 16 pair successive states with spherical PCA projections of state representations and inferred waypoints.Source: §B.3 · p. 23, Figure 15 · p. 28, Figure 16 · p. 30
  • In maze visualizations, inferred latent waypoints correspond to intermediate positions along the path toward the goal.The plotted waypoint positions are nearest neighbors of encoded grid locations; score heatmaps proxy policy reachability.Source: §B.3 · p. 23, Figure 17 · p. 32

C EXPERIMENTAL DETAILS

  • The implementation uses JaxGCRL and BuilderBench codebases, with training curves averaged over five seeds and runs taking 12–72 hours depending on the environment.The shaded regions denote one standard error.Source: C EXPERIMENTAL DETAILS · p. 33

C.1 ENVIRONMENTS

  • Experiments use JaxGCRL navigation tasks and BuilderBench cube-manipulation tasks, including modified BuilderBench with direct cube actuation instead of a gripper.The JaxGCRL set includes Humanoid, Humanoid U Maze, Ant Big Maze, Ant U4 Maze, Ant U5 Maze, and Ant Extreme Maze.Source: §C.1 · pp. 33–34
  • Success is measured as the fraction of evaluation episodes meeting task-specific criteria: torso distance in JaxGCRL and cube placement and structure-position tolerances in BuilderBench.The four-cube pyramid additionally requires its horizontal cube arrangement to match the target within specified tolerances.Source: §C.1 · p. 34

C.2 IMPLEMENTATION DETAILS

C-Slerp periodically selects a waypoint count and subgoal for a low-level policy, then uses collected transitions to update its representations and policy. The implementation also specifies training settings and baseline configurations for comparison.

C.2 IMPLEMENTATION DETAILS

  • Training samples waypoint counts from a softmax with temperature η = 0.1 and divides fH and fL by τNCE = 0.1 for the InfoNCE loss.Entropy coefficients are automatically tuned; BuilderBench uses repetition factor 12 for in-trajectory negatives.Source: §C.2 · p. 34
  • Every h environment steps, the high-level planner selects the waypoint count and subgoal for the low-level policy to pursue during the next h steps.Transitions are stored in replay and used to update the representations and policy.Source: §C.2 · pp. 34, 36
  • Baselines share the training and evaluation framework, with shared backbone choices where appropriate; HIQL retains component-specific networks and smaller policy layers.HIQL uses a 10-dimensional latent subgoal representation, while L3P uses a 16-dimensional latent space.Source: §C.2 · pp. 34–35
  • CRL follows benchmark-codebase implementations, RIS uses diagonal Laplace subgoals and goal relabeling, and SAC+HER relabels goals from future achieved states or the rollout goal.RIS uses imagined-subgoal samples during training and conditions directly on the task goal at evaluation; SAC+HER recomputes rewards for relabeled goals.Source: §C.2 · pp. 34–35
  • Online HIQL replaces offline data and pretraining with online trajectories, while L3P plans over learned and replay-based landmarks during training and learned landmarks at evaluation.HIQL uses target horizons K = 25 in JaxGCRL and K = 10 in BuilderBench; L3P refreshes its graph every 5 × 10^5 interactions.Source: §C.2 · pp. 34–35
  • Baseline configurations start from original papers and implementations, are tuned per baseline and benchmark, then fixed for final evaluation.Candidate tuning ranges and selected method-specific parameters are reported in Tables 2–7.Source: §C.2 · p. 35, Table 2 · p. 37

Related papers

Loading 2610.20409v1…