Source-linked AI summary
Mollified Value Learning
Hrishikesh Viswanath, Juanwu Lu, S. Talha Bukhari, Mihir Chauhan, Damon Conover, Ziran Wang, Aniket Bera
TL;DR
Offline GCRL needs accurate value estimates despite limited state-action coverage, while pointwise physics-informed constraints can be unstable in complex, high-dimensional settings. MVL replaces those constraints with local spatial expectations that mollify value geometry, and experiments report structured representations with improved goal-reaching performance. The method is nevertheless limited by Euclidean, isotropic-kernel assumptions and approximates rather than exactly solves HJB constraints.
Problem
Offline GCRL must estimate structured goal-conditioned values from static datasets despite limited state-action coverage and weaknesses in temporal-difference learning.
Method
MVL reframes shortest-path value consistency as an expectation over local spatial distributions, creating a tractable geometric regularizer without explicit differential operators.
Results
MVL learns structured value representations and improves goal-reaching performance across robotic navigation and high-dimensional manipulation tasks.
Takeaways & Limitations
Localized value consistency provides a simple, tractable geometric inductive bias for offline GCRL, including high-dimensional manipulation tasks.
Takeaways & Limitations
MVL uses Euclidean distance and isotropic Gaussian kernels, approximating distance constraints rather than exactly representing the HJB Hamiltonian and potentially excluding optimal solutions for some MDPs.
Abstract
from arXiv · showhide
Offline goal-conditioned reinforcement learning (GCRL) learns goal-reaching behaviors from static datasets, but accurate value estimation remains challenging under limited state-action coverage. Existing physics-informed approaches address this by imposing pointwise distance-like geometric constraints derived from Hamilton--Jacobi--Bellman (HJB) optimality principles, often through first-order partial differential equations such as the Eikonal equation. However, enforcing local consistency through explicit differential structure can become unstable in complex, high-dimensional environments. Our key insight is to instead reinterpret distance-like constraints as an expectation over a local spatial measure. By aggregating constraints over this measure rather than evaluating them pointwise, the objective acts as a spatial mollifier, inducing distance-like value geometry without requiring expensive differential operators. We refer to this as Mollified Value Learning (MVL). Experiments across navigation and high-dimensional robotic manipulation tasks show that MVL learns structured, value representations, improving goal-reaching performance, when used with implicit value representation learning methods. Open-source codes are available at https://github.com/HrishikeshVish/MVL.
1 Introduction
Offline GCRL must estimate goal-conditioned values from static data, but temporal-difference learning can miss goal-reaching geometry and produce poorly structured estimates. MVL addresses this by replacing pointwise geometric constraints with local spatial expectations, yielding structured values and improved goal-reaching performance.
- Offline GCRL learns goal-reaching behaviors from static datasets, with the goal-conditioned value function supporting policy extraction and planning.
- Temporal-difference updates lack structural inductive biases for goal-reaching geometry, especially in sparse-reward settings.
- Physics-informed approaches impose geometric constraints from optimal control through Eikonal and Hamilton–Jacobi–Bellman partial differential equations.
- MVL reframes shortest-path geometry as an expectation over a local spatial measure, replacing pointwise derivatives with a tractable integral and smoothing value geometry.
- MVL learns structured value representations and improves goal-reaching performance across challenging robotic navigation and manipulation tasks.
2 Related Works
Offline GCRL methods address long-horizon value errors and geometric structure through several architectural and estimation strategies. Physics-informed value learning adds Eikonal or HJB constraints, but first-order formulations can become unstable in complex, high-dimensional problems.
- Offline GCRL methods include expectile regression, hierarchical decomposition, horizon reduction, quasimetric architectures, and contrastive learning.
- Implicit value methods enable transitive planning without out-of-sample action queries but can suffer from compounding value errors over long trajectories.
- Most existing methods learn values without explicit physics priors, risking biased estimates from bootstrapping and poor generalization under limited state-action coverage.
- Physics-informed approaches derive value-learning objectives from HJB equations or constrain value gradients using Eikonal regularization.
- First-order geometric constraints can be ill-posed for high-dimensional, complex problems, causing numerical instability in neural approximations.
3 Preliminaries
The preliminaries formulate GCRL and its optimal-control interpretation using continuous-time dynamics, trajectories, costs, and goal-conditioned value functions. The value function is represented as negative optimal cost-to-go and obeys a short-step recursion.
- Goal-conditioned Reinforcement Learning: A goal-conditioned MDP comprises state and action spaces, transition dynamics, and a reward depending on the current state and goal.
- Goal-conditioned Reinforcement Learning: GCRL seeks policies that maximize expected cumulative discounted rewards for reaching a goal state.
- Goal-conditioned Reinforcement Learning: Trajectories are sequences of states and actions generated under a policy over a finite decision horizon.
- GCRL with Optimal Control: The continuous-time optimal-control formulation uses dynamical evolution, instantaneous cost, terminal cost, and an expected trajectory objective.
- GCRL with Optimal Control: The goal-conditioned value function is negative optimal cost-to-go and satisfies a small-time recursion involving future value and accumulated cost.
4 Mollified Value Learning
MVL converts shortest-path value geometry into a local, expectation-based consistency objective. Under compactness, deterministic Lipschitz dynamics, and unit running cost, it samples nearby states and penalizes violations with an empirical mollified regularizer.
- 4 Mollified Value Learning: MVL extends shortest-path value bounds across continuous neighborhoods, producing a region-level geometric objective approximated from sampled transitions.
- 4.1 The Idea: The formulation assumes compact state and action spaces, deterministic dynamics that are Lipschitz in state, and unit running cost.
- 4.1 The Idea: Under these assumptions, optimal value corresponds to the shortest feasible path, while small-displacement transition cost is approximated by spatial distance.
- 4.1 The Idea: Shortest-path consistency is relaxed into an expectation over local state distributions rather than enforced pointwise.
- 4.1 The Idea: The integral formulation uses a local transition kernel to aggregate distance-like constraints and produces a gradient structure similar to an Eikonal value field in a toy obstacle problem.
- 4.1 The Idea: The practical regularizer samples nearby states from a local distribution, approximates the expectation with finitely many samples, and applies a ReLU-based empirical objective.
- 4.1 The Idea: MVL penalizes anomalously sharp local inconsistencies while remaining inactive when local shortest-path consistency already holds.
5 Experiments
Experiments evaluate MVL across state-based navigation and manipulation benchmarks, qualitative value geometry, robustness, and real-world robotic tasks. The results show improved geometric structure and goal-reaching performance, while the method remains bounded by Euclidean and isotropic-kernel assumptions.
- Experimental setup: The evaluation covers 13 OGBench state-based tasks spanning navigation and manipulation, plus Franka Kitchen and two custom 7-DoF Panda tasks.The navigation tasks include pointmaze, antmaze, and humanoidmaze; manipulation tasks include cube, scene, and puzzle.
- Baselines: MVL extends GCIVL and is compared with representation strategies including VIB, VIP, TRA, BYOL, and Eikonal constraints.Additional quasimetric baselines are evaluated in an oracle-representation setting.
- Value geometry: On PointMaze-Large, MVL produces goal-aligned, less jittery, and more geometrically structured value contours than corresponding baseline representations.The qualitative ablation compares Original, Dual, VIB, and Eikonal configurations.
- Robustness: Under severe transition noise, MVL recovers success from 0 to 99 on cube-single-noisy and from 0 to 53 on scene-noisy.The authors attribute this robustness to evaluating local spatial distributions rather than fitting individual noisy transitions.
- Real-world evaluation: In Reach and Grasp, success improves from 2/10 without regularization to 8/10 with it, while Pick and Place reaches 10/10 with regularization.The unregularized Pick and Place policy consistently fails to reach goal proximity.
- Limitations: MVL uses Euclidean distance and isotropic Gaussian kernels, approximating distance constraints rather than exactly solving the HJB Hamiltonian.These kernels regularize ambient spatial geometry but do not model admissible dynamics or feasible trajectories.
6 Conclusion
The paper concludes that localized consistency provides a tractable geometric regularizer for goal-conditioned value learning, while identifying representation spaces as an important direction for extension.
- Conclusion: MVL provides a simple, tractable inductive bias that improves value learning in high-dimensional manipulation tasks.The framework is based on localized consistency of value structure.
- Future work: Future work should extend the approach to learned or manifold-aware representation spaces, especially for pixel domains where Euclidean latent-space assumptions may not hold.The conclusion specifically identifies visual representations as a setting where the current geometric assumptions may be insufficient.
A Derivations
The derivation characterizes the goal-conditioned value through shortest-path geometry and the HJB equation. It then connects triangle-inequality constraints and first-order dynamics to local optimality conditions, while noting practical nonsmoothness in robot-learning value functions.
- Assumptions: The analysis assumes compact, convex state and action spaces, Lipschitz dynamics, and bounded nonnegative costs with zero goal cost.The relaxed dynamics are unit-speed and isotropic, with f(s, a) = a and ||a||≤1.
- Assumptions: The optimal value is modeled as the negative of a locally Lipschitz distance-like quasimetric satisfying the triangle inequality.Specifically, V*(x, z) = −d(x, z), with d(x, z) ≤ d(x, y) + d(y, z).
- Practical motivation: In offline GCRL, the HJB Hamiltonian supplies local geometric inductive bias, but robot-learning value functions may develop nonsmooth regions such as shocks or intersecting characteristics.The minimizer of the HJB expression determines an optimal policy under the stated formulation.
- Value inequalities: By optimality and the triangle inequality, valid transitions impose a value relation linking the source, successor, and goal states.For a transition from s to s′, V(s, g) ≥ V(s, s′) + V(s′, g), with a one-step cost bound in the infinitesimal limit.
- HJB derivation: A first-order expansion of the value along system dynamics produces the HJB Hamiltonian, which combines dynamics, value gradients, and cost.The Hamiltonian is H(st, g, ∇V*) = [f(st, at)⊤∇V*(st, g) − c(st, a)].
A.3 Integral Relaxation
The section replaces unstable pointwise geometric constraints with a locally averaged residual over a spatial measure. This mollified formulation yields expected quasimetric consistency and spatial regularization without explicit differential operators.
- Motivation: Pointwise HJB-style constraints can break down at shock boundaries where optimal characteristic curves intersect.The Taylor expansion and HJB formulation may fail at these non-smooth regions.
- Mollified operator: The residual Φ(V, s, s′, g) compares value variation with local cost-scaled displacement, while Rδ[V] aggregates it over neighboring states.The operator is defined on locally regular value functions and uses a one-sided residual decomposition.
- Mollified operator: The mollifier ρδ is a localized probability density whose bandwidth δ controls averaging locality and whose weighting smooths local state geometry.The kernel is normalized and has finite spatial moments.
- Expected consistency: The relaxed constraint bounds aggregate expansive behavior by aggregate contractive behavior, allowing individual neighboring states to violate pointwise consistency.This produces weak expected quasimetric consistency rather than a strict pointwise constraint.
- Expected consistency: The expected quasimetric defect controls the probability of large local violations under the spatial kernel ρδ.This follows by applying Markov’s inequality to the nonnegative expansive residual.
- Regularization: Isotropic averaging adds a higher-order diffusion-like regularization effect to the leading local quasimetric constraint.The local curvature characterization derives this effect through second-order expansion under an isotropic, zero-first-moment kernel.
- Interpretation: The mollified objective regularizes value geometry in ambient state space rather than modeling dynamics-valid transitions or directional feasibility.It complements directional Bellman supervision by controlling sharp oscillations, discontinuities, and locally expansive artifacts.
C Implementation and Hyperparameters
The experiments use sparse-reward offline GCRL with representation-based value learning on a diverse OGBench suite. Evaluation measures final-training success across navigation, manipulation, stochastic pixel environments, and trajectory action distributions.
- Training: The sparse reward is 0 at the goal and −1 otherwise, so the optimal value encodes discounted temporal distance.Dual representations approximate this value through the inner product of learned state and goal representations.
- Benchmark: The benchmark spans geometric navigation, high-dimensional manipulation, and stochastic pixel-based physics tasks.The suite tests varying state dimensionality, transition stochasticity, and dynamic complexity.
- Evaluation: Performance is the average success rate over the final three training epochs, aggregating 50 state-based episodes across five evaluation goals.The reported window covers 800K–1M gradient steps.
- Qualitative comparison: The action distributions align parallel to corridor walls for MVL, identifying a safe path while matching the geometry trend of the Eikonal baseline.The comparison highlights geometry modeling without expensive and unstable first-order automatic differentiation.
D Implementation Details
MVL is implemented as a spatial-expectation objective compatible with accelerator-friendly vectorized computation. Sampling local state distributions avoids explicit spatial derivatives and their associated automatic-differentiation overhead.
- Implementation: The spatial-expectation formulation is compatible with GPUs and TPUs through JAX’s vectorized operations.It evaluates local state distributions instead of explicit spatial derivatives.
- Implementation: Avoiding explicit derivatives removes sequential bottlenecks and graph-unrolling overhead typical of automatic differentiation for continuous constraints.The paper also provides a minimal algorithm for the mollified geometric objective.
E Ablations
The ablations vary neighborhood size and spatial radius to assess MVL’s sensitivity. Performance improves with more neighborhood samples up to saturation, while sufficiently small radii cause a marked drop.
- Ablation setup: The ablations vary neighborhood size K and radius δ, using contour plots of the learned value function and results in Table 3b.The objective samples local perturbations around each state-goal pair and aggregates their residuals.
- Neighborhood size: 57% to 77%: increasing K from 1 to 5 samples improves performance, after which the success rate saturates.This result is reported for the ablation in Table 3b.
- Additional evaluations: The study also reports success-rate ablations over sampled states N and spatial variance δ across four environments with the largest performance gains.Additional result tables cover powderworld, kitchen, and oracle-representation variants.
- Spatial radius: δ ≤10^-3 causes a marked performance drop, indicating greater sensitivity to the spatial radius than to neighborhood sample count.The cited result specifically identifies lower radius values as the problematic regime.