PaperlayerView exact revision on arXiv

Source-linked AI summary

QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents

Sergio Hernández-Gutiérrez, Matteo Merler, Ilze Amanda Auzina, Joschka Strüber, Ameya Prabhu, Matthias Bethge

arXiv:2606.32034v1cs.LGcs.AIcs.CL

TL;DR

Long-horizon agents need denser guidance than outcome-only rewards provide, but existing evaluations are expensive and confounded by training-pipeline choices. QVAL introduces a training-free testbed that compares signals by Q-alignment to reference-policy values before training. Across its benchmark, simple prompting is strongest and methods cluster by family, while the framework remains a pre-training diagnostic rather than a replacement for full post-training evaluation.

  • Problem

    Existing dense-supervision evaluations require expensive downstream training runs and confound signal quality with other post-training engineering choices.

  • Method

    QVAL scores state-action pairs and measures how well each method’s predictions rank them according to estimated Q-values from a reference policy.

  • Results

    Simple direct prompting provides the strongest signal, with methods clustering by family across model sizes, environments, modalities, and target types.

  • Takeaways & Limitations

    QVAL provides a cheap diagnostic for filtering dense-supervision candidates by Q-alignment before expensive training runs.

  • Takeaways & Limitations

    Q-alignment does not capture all factors affecting RL post-training effectiveness, including optimization, algorithmic choices, loss integration, and interactions with other signals.

Abstract

from arXiv · show

LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions. In these settings, outcome-only rewards provide too sparse guidance, failing to inform the model about the goodness of intermediate actions. Dense supervision methods aim to solve this problem by scoring intermediate steps, from intrinsic confidence to self-distillation and embedding similarities. However, it is common practice to evaluate them by measuring the downstream performance of a training pipeline that integrates them. This is expensive, conflates supervision quality with training engineering confounders, and renders different methodological families requiring distinct training setups incomparable. As a result, dense supervision methods are rarely benchmarked on common ground. We introduce QVal, a training-free testbed for directly evaluating dense supervision signals. Given a state-action pair, QVal measures how well a method's score is Q-aligned: whether it orders actions according to the Q-values of a strong reference-policy. This lets us compare signals before any training run and separate signal quality from other engineering choices. We instantiate QVal as QVal-v1.0, benchmarking 21 dense supervision methods across four diverse environments and seven methodological families, with over 1.2K evaluation experiments across six open-weight model backbones. We find that simple prompting baselines consistently outperform recent dense supervision methods from the literature, and that performance clusters strongly by family. These findings hold across model sizes, environments, and observation modalities. QVal is designed to be easily extensible to new environments and methods, enabling researchers to iterate on dense supervision methods before any training run.

1 Introduction

QVAL addresses the costly, confounded evaluation of dense supervision by providing a training-free benchmark that compares signals through Q-alignment before downstream training. QVAL-v1.0 applies this framework across diverse environments, methods, and backbones, finding strong performance from simple methods and family-level clustering.

  • Motivation: Long-horizon agents face sparse-reward learning because outcome rewards provide little guidance about intermediate actions and may never be observed.
  • Motivation: Downstream post-training evaluations are expensive and conflate supervision quality with algorithmic, optimization, normalization, loss-integration, and signal-balancing choices.
  • QVAL: QVAL evaluates dense signals before training by measuring whether predicted scores order state-action pairs according to reference-policy Q-values.
  • Findings: Direct prompting and ranking perform best on average, performance clusters by methodological family, and code-based methods weaken in more open-ended settings.
  • Benchmark: QVAL-v1.0 benchmarks 21 methods from seven families across four environments and six open-weight backbones, totaling more than 1.2K experiments.
  • Extensibility: QVAL is extensible to new environments and methods, with new methods requiring only one score per state-action pair for direct comparison.

2 QVAL: A Training-Free Testbed for Dense Supervision Methods

QVAL defines dense-signal quality by how consistently scores rank actions according to expected returns under a reference policy. It uses common inputs and targets, rank-based evaluation, and reference-policy labels to isolate signal quality cheaply.

  • Motivation: QVAL asks whether a dense signal assigns higher scores to actions that make eventual success more likely.
  • Reference values: The Q-value Qπ(s, a) is the expected return after taking action a in state s and then continuing under policy π.
  • Reference policy: QVAL uses a fixed reference policy π to label state-action pairs, while the policy ultimately trained with the signal may differ.
  • Q-alignment: A signal k is Q-aligned when it orders state-action pairs consistently with their reference values Qπ, with perfect alignment reproducing the same ranking.
  • Interpretation: Q-alignment is intended as a cheap proxy for downstream usefulness when the reference policy closely approximates an optimal policy.
  • Evaluation: QVAL estimates alignment with rank correlation because methods produce scores on incompatible scales; Spearman’s ρ is the main metric and Kendall’s τ is reported separately.

3 QVAL-v1.0: Benchmarking Dense Supervision Methods

QVAL-v1.0 benchmarks dense supervision methods across varied environments, modalities, and methodological families using a common evaluation setup. It organizes methods by the information used to score state-action pairs and standardizes their comparison through reference-policy labels and Q-alignment metrics.

  • Environments: QVAL-v1.0 initially covers four environments spanning navigation, embodied reasoning, computer use, and terminal-based problem solving.The suite varies action-space structure, observation modality, and the context needed to evaluate an action.
  • Data collection and labeling: Each environment contributes state-action pairs labeled with reference-policy Q-values, enabling direct evaluation of predicted action scores.Labels are obtained by forcing the dataset action and estimating discounted continuation returns, using optimal policies, an expert planner, or MVMC rollouts depending on the environment.
  • Method taxonomy: The benchmark evaluates 21 dense supervision methods grouped into seven families according to the information used to score state-action pairs.The families include direct prompting, intrinsic signals, code generation, self-distillation, ranking, pretrained models, and embedding similarity.
  • Direct and ranking methods: Ranking methods directly compare candidate actions from the same state, while direct methods prompt models to output numeric values for datapoints.Direct variants differ in whether they score one or multiple datapoints per prompt, whereas ranking produces an action permutation.
  • Intrinsic and self-distillation methods: Intrinsic methods use model confidence or rubric-based judgments, whereas self-distillation uses changes in action likelihood after privileged outcome information is revealed.The self-distillation family includes sdpo using the next state and sdpo-gt, which additionally reveals expert-action and trajectory information.
  • Code, embedding, and pretrained methods: Code methods generate executable scoring functions, while embedding and pretrained methods use fixed visual or vision-language representations to score states.Codegen variants differ in independent function generation and prediction averaging; embedding methods compare state and goal representations.
  • Backbones and evaluation: The evaluation fixes the LLM or VLM backbone across methods wherever possible so Q-alignment differences reflect scoring methods rather than underlying model performance.The benchmark prioritizes open-weight backbones with accessible hidden states or token log probabilities.

4 Results

QVAL reveals that simple direct-prompting and ranking methods achieve the strongest Q-alignment, while performance varies substantially across methodological families, environments, modalities, and value targets. Robustness analyses show that added complexity and visual input generally do not improve alignment, although conclusions remain stable across reference backbones.

  • 4.1 Main Results: Direct prompting and ranking methods achieve the highest Q-alignment across environments and model backbones, with methods clustering by methodological family.Code-based methods show the largest variance, performing better in structured environments and weakening in open-ended settings.
  • 4.1 Main Results: Within-family complexity rarely improves Q-alignment: elaborate direct, code-generation, and self-distillation variants do not reliably outperform simpler alternatives.Codegen-avg improves mean correlation slightly over codegen but retains substantial variance, while sdpo-gt does not improve over sdpo.
  • 4.1 Main Results: Q-alignment does not decline monotonically with task difficulty; direct-prompting methods remain positive across environments, while other families respond differently to environment characteristics.Code and ranking weaken in open-ended environments, with code becoming negative on TerminalBench, whereas self-distillation performs more strongly there.
  • 4.2 Robustness Analyses: Text representations yield more reliable recovery of reference values than images in the evaluated settings, suggesting that parsing visual information is more challenging.Figure 4 compares method–environment–model combinations by placing text correlations on the x-axis and image correlations on the y-axis.
  • 4.2 Robustness Analyses: Method rankings are largely preserved between Q-values and state values, although absolute correlations differ by family.Code and pre-trained methods align better with state values, while direct prompting aligns better with Q-values.
  • 4.2 Robustness Analyses: TerminalBench method correlations closely match when reference values are estimated with GPT-5.5 or Claude Opus 4.7, supporting robustness across independent frontier-model policies.Positive correlations under one reference backbone generally remain similarly positive under the other.

5 Related Works

Prior evaluations embed dense supervision signals in downstream training or selection pipelines, while reward-model benchmarks use direct comparison tasks. These approaches demonstrate utility but make signal quality difficult to isolate across methods.

  • Downstream training evaluations report task-return or pass-rate improvements after integrating dense feedback into post-training or selection pipelines.Examples include self-evaluation signals for reasoning search and generative verifiers assessed under fixed inference budgets.
  • Measured gains depend on policy models, optimizers, exploration, environment distributions, generated-data amounts, and training-loop implementation details.
  • RewardBench, RM-Bench, and RewardBench 2 evaluate reward models through response-comparison or accuracy tasks spanning chat, reasoning, safety, subtle errors, and style biases.

6 Conclusion

QVAL concludes that training-free Q-alignment provides a common, extensible way to benchmark dense supervision methods across environments and models. The release documents the benchmark’s initial environments, methods, and supporting materials.

  • QVAL-v1.0 benchmarks 21 dense supervision methods from seven families across four environments and six open-weight backbones.
  • Simple direct prompting provides the strongest signal, while methods cluster reliably by family across model sizes, environments, modalities, and target types.
  • New methods can be added by emitting a single score per state-action pair, and researchers can use QVAL to build datasets for their own tasks.
  • The paper provides detailed environment, metric, method, and model-configuration descriptions alongside complete results summarized in tables.

A.1 TerminalBench

QVAL-v1.0 spans four environments with distinct observations, actions, rewards, collection procedures, and reference policies. The appendix details these configurations, including practical choices for generating reliable reference labels.

  • A.1 TerminalBench: TerminalBench uses text-only terminal observations, open-ended shell commands, binary verifier rewards, and DeepSeek v3.2 trajectories collected under bounded episode and command horizons.The easy TBLite split contains 19 tasks because many full-TerminalBench tasks are unsolvable by current models, often due to timeouts.
  • A.1 TerminalBench: TerminalBench labels use Max-Value Monte Carlo with GPT-5.5 and k = 16 rollouts, achieving Pass@16 of 100% on the selected task subset.The reference choice is intended to avoid convoluted single-command solutions associated with an optimal policy.
  • A.2 OpenApps: OpenApps contains eight synthetic web-application tasks with AXTree or screenshot observations, BrowserGym actions, and server-checked goal rewards.Its scripted collection policy mixes ε=0.25 random actions and retains 94 evaluation and ranking triples.
  • A.3 ALFWorld: ALFWorld exposes household tasks through text or rendered images, but visual expert-planner quality degrades on five of six task types, so the scope is restricted for uniform labels.The handcoded planner uses simulator PDDL state and agrees with DeepSeek-v3.2 Max-Value MC at Spearman 0.83 on shared points.
  • A.4 FrozenLake: FrozenLake uses deterministic 8×8 maps with ASCII or rendered-grid observations, four discrete moves, sparse goal rewards, and a shortest-path reference policy.Eight random maps are cycled across trajectories, while ε=0.1 collection noise yields approximately 34% trajectory success.

B Evaluation Metrics

QVAL evaluates dense signals by rank correlation because methods produce incompatible numerical scales and practical use depends on induced ordering. It reports global and within-state ranking metrics with complementary sensitivities.

  • QVAL compares predicted values with reference labels using rank-based correlations rather than absolute error.Rank transformations make the comparison insensitive to incompatible scales such as LLM scores, code outputs, and token log-probabilities.
  • Spearman’s ρ is Pearson correlation between ranks and is invariant under monotonic transformations of either variable.
  • Kendall’s τ counts concordant and discordant pairs, treating each pairwise inversion equally and handling ties through label and prediction tie counts.
  • Both metrics range from −1 to 1; Spearman emphasizes squared rank errors, whereas Kendall is more robust to extreme outliers and typically has smaller absolute values.
  • Failed predictions are recorded as NaN and omitted before correlation, while experiments retain enough non-NaN points for high statistical significance.
  • Ranking methods produce a permutation over candidate actions and are scored per state, then averaged across states with non-degenerate label rankings.This isolates within-state action discrimination from cross-state value calibration.

C Method Details

This appendix documents the configurations, prompt templates, and released launch files for every evaluated method.

  • The appendix specifies each evaluated method’s parameterisation and execution details.It organizes shared prompt context, method families, prompt templates, and experiment configuration files.

C.1 Shared method context

Prompt-based methods share a common environment context but differ in how they elicit, aggregate, or derive value scores. The appendix also records method-specific input requirements and ranking procedures.

  • Every prompt-based method receives task, reward, space, state, action, next-state, and bounded-history information through a shared MethodContext.The next state is included for Q-value and shaped-reward signals.
  • Direct variants differ in batching: single-point, packed four-point, sequential up-to-eight-point, or 16-sample averaging.The 16-sample variant averages parsed responses to separate signal quality from sampling noise.
  • GVL estimates per-state values from a shuffled full trajectory, while logprob readout scores normalized Yes-versus-No probabilities.The logprob method requires raw token logprobs, excluding backbones without that capability.
  • Verifier scores average normalized logprob responses across environment-specific criteria, with points randomly grouped to remove trajectory-order artifacts.The criteria cover correctness, efficiency, error detection, and, for ALFWorld, precondition awareness.
  • ΔBelief compares pre-action and post-action success beliefs, whereas codegen samples executable value functions and reports both per-sample and averaged correlations.SDPO instead ranks candidate actions using teacher–student log-probability differences, with SDPO-GT providing an oracle-teacher variant.
  • Eureka iteratively searches over candidate functions, retaining the best function according to an LLM judge on held-out points.The search uses 16 outer iterations, eight candidates per iteration, and eight judge points.

C.8 Embedding-similarity (VLE) methods

The VLE methods use frozen image-text encoders to compare state images with target and negative goal descriptions, using either continuous or thresholded scores.

  • All VLE variants use frozen CLIP ViT-L/14 and SigLIP-base encoders with independently encoded state and goal text.Scores are computed as cosine similarity in the joint embedding space.
  • Goal descriptions are fixed by environment, using a target phrase or the task-specific objective supplied by the simulator or wrapper.FrozenLake uses a fixed elf-and-gift-box target description.
  • vlm-rm-cos scores raw state–goal cosine similarity, while vlm-rm shifts embeddings along a goal-minus-baseline direction with α = 0.5.The latter adds an environment-specific generic-state baseline.
  • vlm-sor-softmax assigns a continuous target-goal probability at τ = 0.07, whereas vlm-sor thresholds that probability at β = 0.5 into binary reward.Both variants are retained to expose information loss from thresholding.

C.9 Pre-trained value methods

Pre-trained and prompt-based value methods estimate state or action value through visual embeddings, trajectory context, verifier bins, or direct Q-value prompts. Their implementations vary in goal representation, temporal context, and output format.

  • VIP uses image-only ResNet-50 embeddings and negative L2 distance to a goal image taken from the reference rollout’s final state.The goal image uses the trajectory_end setting.
  • LIV supports image-goal cosine or negative-L2 scores and a text-goal cosine score using its corresponding embedding towers.Image-goal variants use the same reference-rollout final-state source as VIP.
  • For encoder methods, action value is evaluated by scoring the next state as ˆQ(s, a) = ˆV(s′).
  • The benchmark records method-specific hyperparameters and model-specific decoding settings, with sample counts defined per evaluation point unless stated otherwise.
  • Direct prompting supplies history, current state, action, and next state, then requests a single numeric Q-value under an optimal-play assumption.Batched and sequential wrappers alter how multiple datapoints are presented and returned.
  • GVL provides shuffled surrounding transitions as context and asks for a value only for a final target transition whose next state is shown.
  • Verifier methods map judgments onto an ordered A–T score scale, with higher bins indicating stronger evidence that the action and resulting state are favorable.The verifier can also condition judgments on environment-specific evaluation criteria.

E Complete Results

The complete-results tables report correlations for Q-value and state-value signals across environments, modalities, and label-generation settings. They include TerminalBench, OpenApps, ALFWorld, and FrozenLake, with text and vision slices where applicable.

  • Table conventions: The tables report Spearman (ρ) or Kendall (τ) correlations between predicted and ground-truth signals across evaluation points.Significance markers indicate p < .10, p < .01, and p < .001; codegen aggregate cells additionally report mean ± std and [min, max].
  • Schema conventions: Rows, metric rows, and model columns without values are omitted, while conceptually inapplicable slices are excluded from the table schema.Examples include ranking or belief metrics in V-value tables and CLIP or SIGLIP columns in vision tables.
  • TerminalBench: TerminalBench tables cover Q-value correlations with text observations using Codex 5.5 and Opus 4.7 Max-Value Monte Carlo label generation.The supplied table entries identify the environment, value target, modality, and rollout backbone but do not provide cell values.
  • OpenApps: OpenApps tables cover Q-value correlations in text and vision, plus state-value correlations in text.These slices use a scripted policy with Max-Value Monte Carlo for label generation.
  • ALFWorld and FrozenLake: ALFWorld and FrozenLake tables span Q-value and state-value targets across text and vision modalities where listed.The tables use a scripted policy with Max-Value Monte Carlo for label generation.
Loading 2606.32034v1…