PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Value Classifier

Alexander Swerdlow, Perry Dong, Chelsea Finn

arXiv:2610.22497v1cs.LG

TL;DR

Value learning faces challenges in sparse-reward settings, and conventional return-based supervision leaves actions unused. GVC learns an action model conditioned on state and return, with value anchoring to bootstrapped targets; it reports strong performance relative to prior methods on online and offline tasks.

  • Problem

    Sparse-reward value learning is challenging, and conventional scalar or distributional return supervision does not use actions as an additional source of learning signal.

    Source: §1 · pp. 1–2

  • Method

    GVC models action likelihood conditioned on state and return, recovers a return distribution with Bayes’ rule, and anchors Q-values to bootstrapped targets.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    GVC reports strong performance relative to prior methods across online and offline tasks; on Tool Hang it reaches roughly 85% success within 0.5M steps, versus about 50% for QC and RLPD at 1M steps.

    Source: §1 · p. 2, §5.1 · p. 8

  • Takeaways & Limitations

    The reported analysis finds that removing generative supervision correlates with lower performance on the same architecture.

    Source: §1 · p. 2

    GVC requires T forward passes over K return bins in theory, and its factorized components and discretized return space increase implementation complexity.

    Source: §6 · p. 11

Abstract

from arXiv · show

Q-learning is premised on learning a function to predict the value of a state-action pair, with the goal of using such a function to guide policy improvement. Most approaches learn this function from scratch using scalar or distributional return targets. This discriminative approach leaves another source of supervision for learning representations unused: the actions themselves. Rather than directly predict the return of a state-action pair, we ask the reverse question: how likely is the action under each possible return? This perspective provides a critic whose generative component is a return-conditioned policy, which can share the actor's architecture. Training this policy supervises the critic with every action rather than with returns alone, and a squared-error loss anchors the critic's Q-value to the bootstrapped target. We evaluate our approach on both offline and online RL tasks and show gains over leading discriminative methods, including distributional RL methods.

1. Introduction

Value functions can support sample-efficient policy improvement, but learning them remains difficult in sparse-reward settings. GVC uses actions as supervision by modeling their likelihood under possible returns, then anchors its value estimate to a bootstrapped target.

1. Introduction

  • Sparse rewards intensify value-learning challenges: Monte Carlo estimates have high variance, while TD bootstrapping can compound errors across long trajectories.Both approaches can require additional rollouts for accurate estimation.Source: §1 · pp. 1–2
  • Prior approaches add representational capacity or inductive biases through distributional-return prediction or contrastive future-state occupancy prediction; GVC instead asks whether actions can provide supervision.Distributional RL models the return distribution, while contrastive methods predict future-state occupancy.Source: §1 · p. 2
  • GVC models the likelihood of an action given a state and return, using Bayes’ rule and a return prior to recover a return distribution and its mean Q-value.A squared-error value-anchoring loss fits Q(s, a) to a bootstrapped target while retaining generative supervision.Source: §1 · p. 2
  • GVC combines return-conditioned flow matching with value anchoring and is evaluated on online RoboMimic and offline and offline-to-online OGBench tasks.The reported analysis says removing generative supervision correlates with lower performance on the same architecture.Source: §1 · p. 2

2. Related Work

Related work applies generative models to discriminative tasks and develops increasingly expressive value-function models. GVC reverses the usual value-prediction direction, scoring actions under return conditions and recovering Q-values with Bayes’ rule.

2. Related Work

  • Generative classifiers and hybrid generative–discriminative objectives motivate using generative models for discriminative tasks; GVC treats discretized returns as classes and actions as generated outputs.The paper positions this as an implicit Q-function for reinforcement learning.Source: §2 · p. 2
  • Unlike robotics value models that cannot judge policy actions directly, GVC scores actions from a generative policy conditioned on return.The paper contrasts this with language models that can express reward or grading responses in text.Source: §2 · p. 3
  • Value-learning methods vary in value representation and network architecture but predict returns from state-action pairs; GVC instead models actions given state and return, unlike Decision Transformer’s MC-return conditioning and lack of Bellman backups.GVC uses bootstrapped Bellman targets and its action model to score candidate actions.Source: §2 · p. 3

3. Preliminaries

The preliminaries specify the discounted-return and off-policy TD setting, then introduce flow matching and its use for classification through class-conditional noising errors.

3. Preliminaries

  • An MDP is specified by state and action spaces, transitions, rewards, an initial-state distribution, and a discount factor; its action-value function is expected discounted return conditional on state and action.The paper approximates the return distribution with categorical support points called return bins.Source: §3 · p. 3
  • The study focuses on off-policy Q-learning from replay-buffer transitions, with TD targets using a frozen target critic and the next action selected by the RL policy.Terminal transitions mask bootstrapping.Source: §3 · pp. 3–4
  • Flow matching transforms Gaussian noise into conditional data along a flow-time path, with sampling integrating the learned velocity from noise to data.The flow time is sampled uniformly on [0, 1].Source: §3 · p. 4
  • A flow classifier estimates a likelihood-like energy by noising a fixed observation under each class; with a uniform class prior, it selects the class with the smallest noising error.The energy is not an exact normalizing-flow density, and the same noise–time pairs are reused across classes.Source: §3 · p. 4

4. Generative Value Classifier

GVC scores actions by modeling their likelihood under discretized returns and combining those likelihoods with a state-only return prior; the posterior mean supplies Q(s, a). An actor proposes candidate actions, while flow-likelihood and value-anchor objectives train the critic.

4. Generative Value Classifier

  • GVC reverses value prediction: it models how likely an action is under each return, combines that likelihood with a state-only return prior, and uses the posterior mean as Q(s, a).At inference, the actor proposes candidates and the critic ranks them by Q(s, a).Source: §4 · p. 4, Figure 2 · p. 5

4.1. From discriminative values to a generative critic

  • Conditioning on state, GVC applies Bayes’ rule to return-conditioned action likelihoods and a return prior to recover the return distribution given an action.Conditioning on state avoids requiring a generative model of states.Source: §4.1 · p. 5

4.2. Flow likelihood

  • GVC estimates each return-bin action likelihood using conditional flow-matching error as a surrogate for negative log-likelihood, with tunable weights for likelihood-like factors.The likelihood scores use multiple stratified flow timesteps and antithetic noise to reduce variance.Source: §4.2 · pp. 5–6

4.3. Objectives

  • The critic uses a projected bootstrapped return distribution as a soft target for both its generative loss and return prior, then adds squared-error value anchoring because flow losses alone do not ensure the target mean.The anchor constrains the return used to rank actions.Source: §4.3 · pp. 6–7
  • A behavior-cloning flow actor proposes actions, and evaluation executes the sampled candidate with the largest critic Q-value, replanning at the next step.Best-of-N sampling is used for implicit extraction in the experiments, except for the stated ablation.Source: §4.3 · p. 7

5. Experiments

Experiments compare GVC with methods that vary the critic, actor, or training recipe across RoboMimic and OGBench. GVC performs strongly in offline and offline-to-online OGBench evaluations, while Tool Hang ablations examine which parts of its objective matter.

5. Experiments

  • The experiments ask whether GVC improves online RL with the recipe held fixed, whether gains extend to offline settings, and which objective components matter.They use RoboMimic manipulation tasks and OGBench offline and offline-to-online benchmarks, comparing methods that change the critic, actor, or training approach.Source: §5 · pp. 7–8

5.1. Online RL

  • On Tool Hang, GVC reaches roughly 85% success within 0.5M steps, while QC and RLPD reach about 50% at 1M steps and IDQL does not learn the task.Lift, Can, and Square are eventually solved by GVC, QC, and IDQL, with small sample-efficiency gains for GVC.Source: §5.1 · p. 8

5.2. Offline RL

  • On OGBench offline tasks, GVC reaches 92% on puzzle-4x5 and 48% on puzzle-4x6, versus BFN’s 83% and 22%, and averages 8 points above Value Flows on three puzzle domains.GVC leads BFN by 12 points on sparse scene tasks and 4 on cube-triple, but the two are nearly tied on cube-double.Source: Table 1 · p. 9, §5.2 · p. 9

5.3. Offline-to-online RL

  • GVC’s average success rises from 67.1% offline to 98.9% after online finetuning, about 19 points above Value Flows and 26 above FQL.It finishes within 95% of the best method on all six tasks.Source: Table 2 · p. 10, §5.3 · p. 10

5.4. Ablations

  • On Tool Hang, success falls from about 85% at flow-loss weight 1 to 45% at 0.001 or with the loss removed; uniform return-bin weighting reaches about 70%.With shared noise, T=1 reaches about 80%; without sharing, it reaches about 60%.Source: §5.4 · p. 10, Figure 5 · p. 11

6. Discussion

GVC uses a return-conditioned generative likelihood to model the return distribution implicitly. The discussion situates this approach in sample-efficient value learning and notes its compute and implementation costs.

6. Discussion

  • GVC uses a return-conditioned generative likelihood to implicitly model the return distribution.The method extends generative classifiers to a domain where sample-efficiency is particularly important.Source: §6 · p. 11
  • The critic’s theoretical compute cost includes T forward passes to compute likelihood over K bins, potentially across ensembled copies.In practice, the authors reduce this to two forward passes per timestep and report costs comparable to other performant distributional RL methods.Source: §6 · p. 11
  • The presented form factorizes GVC into two model components and discretizes the return space, increasing implementation complexity.Source: §6 · p. 11

A. Additional Ablations

Additional ablations extend the main Tool Hang experiments with Square results and two further comparisons.

A. Additional Ablations

A.1. Ablations on Square

Square shows little separation among flow-loss variants, while value anchoring is essential across the tested tasks and its useful weight range varies by task.

A.1. Ablations on Square

  • On Square, flow-loss variants, including removing the loss, end near 95%, while scoring without shared noise ends several points lower.The default learns slightly faster early; with shared noise, T=1 learns more slowly than T≥2 but catches up.Source: Figure 6 · p. 17, §A.1 · p. 17

A.2. Value-anchor coefficient

  • On Tool Hang, 0.1× anchoring ends near the 1× default (88% vs. 86%), while 10× ends at 77% and 0.01× at 57%.The 10× setting learns more slowly; 0.01× is slower still.Source: §A.2 · p. 17
  • Without value anchoring, success stays below 10% on Square, Tool Hang, and offline puzzle-4x4.With a nonzero anchor, Square is insensitive to its weight, and puzzle-4x4 settings end between 30% and 38% after 106 updates.Source: §A.2 · p. 17, Figure 7 · p. 18

A.3. Best-of-𝑁selection versus a distilled actor

The paper compares best-of-N action selection with a one-step actor trained using direct Q-gradients. Distillation matches best-of-N only on Square, while best-of-N performs better on Tool Hang and puzzle-4x4.

A.3. Best-of-N selection versus a distilled actor

  • On Square, the distilled actor reaches 96% by 106 steps but learns more slowly, reaching 78–90% at 5 × 10^5 steps versus 96% for best-of-N.The one-step actor is trained by direct Q-gradients under GVC and replaces best-of-N selection.Source: §A.3 · p. 18, Figure 8 · p. 18
  • On Tool Hang, best-of-N reaches 91%, while the distilled actors stay near zero for every tested coefficient.The sweep tests α values from 0 to 1000.Source: §A.3 · pp. 18–19, Figure 8 · p. 18
  • On puzzle-4x4, distilled actors peak near 30% around 4 × 10^5 steps and then decline, while best-of-N improves throughout to 40%.The paper attributes the gap to best-of-N staying within behavior-flow support, whereas the distilled actor follows ∇aQ and risks instability.Source: Figure 8 · p. 18, §A.3 · p. 19

B. Compute Cost

GVC scores actions by evaluating return-conditioned likelihoods, which costs more than a scalar critic but is comparable to distributional critics in the reported timing. The appendix describes its shared architecture, scoring, return-bin, loss, and actor-extraction settings.

B. Compute Cost

  • GVC evaluates an action’s likelihood under every return bin rather than predicting a value in one forward pass, using a shared policy architecture and a lightweight head to cover all bins in one pass.Most network capacity goes into learning pθ(a|s); the head converts it to pθ(a|s,z).Source: B. Compute Cost · p. 19
  • GVC takes 15.6 ms per offline puzzle-4x4 gradient update, comparable to Value Flows (13.5 ms) and IQN (13.1 ms), but 5–9× costlier than scalar regression critics (1.8–3.4 ms).The timing is mean and standard deviation over three seeds on one H100.Source: B. Compute Cost · p. 19, Figure 9 · p. 20

C.1. GVC hyperparameters

  • Scoring uses T=4 stratified flow times and antithetic noise, costing 2T forward passes per candidate; noise and times are shared across bins and candidates.The action model returns all K bins per pass; training instead draws one fresh noise–time pair per transition.Source: §C.1 · p. 21
  • Returns are discretized into K uniform bins over environment-specific bounds, and projected targets are smoothed with a Gaussian kernel to discourage collapse to one bin.The shifted target atoms are clipped to the bounds and their mass is linearly interpolated between the two closest bins.Source: §C.1 · p. 21
  • The behavior-cloning flow supplies N candidate actions, and the critic selects the candidate with the highest Q(s,a); βret and βprior separately weight the likelihood and return-prior terms.For ensembles, “mean” averages return distributions bin by bin, while “min” retains the distribution with the lower mean.Source: §C.1 · pp. 20–22, Table 3 · p. 20, Table 4 · p. 21

C.2. Training and evaluation protocol

The protocol specifies distinct online RoboMimic and offline/offline-to-online OGBench training and evaluation procedures. Baselines are tuned on each domain’s default task using the Value Flows protocol, and BFN differs from GVC by replacing its critic.

C.2. Training and evaluation protocol

  • RoboMimic training starts after 5,000 environment steps, uses a 50/50 demonstration–replay batch mix and UTD ratio 1, and evaluates every 25,000 steps.Inference uses five-action open-loop chunks; experiments use three seeds, while the reported ablations use eight.Source: §C.2 · p. 22
  • OGBench offline runs use 10^6 gradient steps, while offline-to-online runs add 10^6 environment steps at UTD ratio 1 and resume updates after 1,000 online steps.The replay buffer begins with the offline dataset; online transitions are added and batches are sampled uniformly from the full buffer.Source: §C.2 · p. 22
  • OGBench uses eight seeds and evaluates over 100 episodes every 10^5 steps; Tables 1 and 2 report means ± sample standard deviations over seeds.Learning curves are smoothed and aggregated as specified separately for Figures 3 and 4; Figures 5–7 use eight seeds per variant.Source: §C.2 · p. 22

C.3. Baseline tuning

  • Baseline hyperparameters are swept on each domain’s default task following the Value Flows tuning protocol; BFN shares GVC’s actor, discount, critic ensemble, and schedule but substitutes a QC regression critic.The BC policy uses four hidden layers of width 512, and CODAC’s conservative penalty is fixed at 0.1.Source: §C.3 · pp. 22–23, Table 5 · p. 23

Related papers

Loading 2610.22497v1…