PaperlayerView exact revision on arXiv

Source-linked AI summary

Specifying Reward Functions for RL Without Environment Sampling

Stephane Hatgis-Kessell, W. Bradley Knox, Emma Brunskill

arXiv:2609.15544v1cs.LGcs.AI

TL;DR

Reward specification through online preference learning is costly when environment interaction is expensive or unsafe. EARS constructs interpretable reward features through a structured LLM process, then learns their weights from preferences over imagined trajectories without environment sampling. Across three long-horizon domains, it produces reward functions more aligned with the ground truth than direct LLM-prompting baselines under both ground-truth and LLM-labeled preferences.

  • Problem

    Online preference-based reward learning can require repeated policy training, real-environment trajectories, and feedback collection, which is costly or unsafe in some domains.

  • Method

    EARS uses structured LLM interaction to construct expressive reward features, then strategically samples imagined trajectories and learns feature weights from preferences over them.

  • Results

    Across three long-horizon domains, EARS produces more aligned reward functions than direct LLM prompting with either ground-truth preferences or LLM-labeled preferences.

  • Takeaways & Limitations

    Preference-based reward specification can remain effective without environment sampling when real trajectory collection is costly or infeasible.

  • Takeaways & Limitations

    The evaluation does not establish whether human stakeholders can meaningfully reason about imagined trajectories that may not be physically realizable.

Abstract

from arXiv · show

Enabling human stakeholders to specify reward functions that lead to their desired outcomes is a key challenge in deploying reinforcement learning agents. Preference-based methods such as online RLHF can reduce the burden of manual reward design, but they require repeatedly training policies, sampling trajectories from the real world, and eliciting feedback, making them impractical in settings where environment interaction is computationally expensive or unsafe. We introduce Experience-Free Autonomous Reward Specification (EARS), a method for learning reward functions from preferences without environment interaction. Our approach uses a structured LLM-mediated process to construct a small set of expressive reward features from a task description and the environment observation space, then strategically samples imagined trajectories in this feature space and learns feature weights from preferences over the imagined trajectory pairs. We evaluate on three long-horizon domains: pandemic lockdown regulation design, insulin administration for diabetes patients, and autonomous vehicle control on a highway. We compare EARS to baselines that also enable reward specification without environment interaction--namely, methods that directly prompt an LLM to generate a reward function. When learning from either ground-truth preference labels or preferences labeled by a LLM, EARS designs reward functions that are more aligned with the ground truth reward function that produced the preferences or LLM context than these baselines. These results suggest that preference-based reward specification remains effective without environment sampling, enabling practical reward design in settings where collecting real trajectories is costly or infeasible.

1 Introduction

Preference-based reward learning can avoid manual reward design, but online methods require costly or risky environment interaction. EARS addresses this by constructing reward features with LLMs and learning their weights from preferences over imagined trajectories.

  • 1 Introduction: Real-world rollouts and diverse suboptimal trajectories can be expensive, unsafe, or infeasible, limiting online and offline preference-based RLHF.These costs include policy training, preference data collection, and acquiring sufficiently diverse experience.
  • 1 Introduction: EARS learns reward functions without sampling environment trajectories by combining structured LLM feature design with preference learning over imagined trajectories.Its two stages construct expressive reward features and then elicit preferences to learn feature weights.
  • 1 Introduction: Across three long-horizon tasks, EARS produces more aligned reward functions than direct LLM prompting when preferences come from ground-truth rewards or LLM labels.The evaluation uses synthetic ground-truth preferences and LLM-labeled preferences based on natural-language task specifications.
  • 1 Introduction: EARS uses zero environment transitions and, with ground-truth preferences, matches the preference-label efficiency of a real-trajectory baseline while outperforming direct prompting.This makes EARS strictly more environment-sample-efficient than online RLHF methods that require environment interaction.
  • 1 Introduction: Directly prompting an LLM with an extremely detailed objective description often does not produce a more aligned reward function than providing no description.This motivates the structured feature-construction and preference-learning process used by EARS.

2 Preliminaries

The preliminaries formulate reward learning in an MDP using feature-based trajectory representations and preference datasets. They describe Bradley–Terry and noiseless-preference approaches for estimating reward weights, while noting the Bradley–Terry assumption can be limiting.

  • 2 Preliminaries: The framework represents the learned reward as a linear function of transition features, while allowing ground-truth and baseline rewards to be nonlinear.EARS learns both the feature representation and its weight vector rather than assuming a known feature basis.
  • 2 Preliminaries: Trajectories are represented by cumulative discounted reward features, and preferences are collected over pairs of these feature-based trajectories.Preference labels encode one trajectory as preferred, the other as preferred, or the pair as equally preferred.
  • 2 Preliminaries: Under the Bradley–Terry model, reward weights are learned by minimizing cross-entropy over preferred trajectory pairs.This models preferences as stochastic comparisons determined by the learned trajectory returns.
  • 2 Preliminaries: The Bradley–Terry assumption is acknowledged as potentially limiting, motivating a separate treatment of noiseless and LLM-labeled preferences.The paper adopts Bradley–Terry for ground-truth preference evaluation while considering alternatives for other labeling settings.
  • 2 Preliminaries: For noiseless preferences, the method instead learns reward weights using constraints or a linear program consistent with the observed preferences.The formulation treats preference labels as generated directly by a reward function rather than by structured noise.

3 Experience Free Automatic Reward Specification (EARS)

EARS uses a two-stage, LLM-mediated process to design interpretable reward features and learn their weights from preferences over imagined trajectories, without sampling the environment.

  • 3.1 Stage 1: Designing a set of reward features: EARS first uses stakeholder and facilitator LLMs to design a small, expressive, interpretable set of reward features from the task and observation space.The features are intended to represent different stakeholder objectives and are later combined linearly.
  • 3.2 Stage 2: Learning weights over the reward features: Stage 2 learns feature weights from preference labels over imagined trajectory pairs, producing the reward function without sampling the environment.The preference labeler may be a human, an LLM, or another source.
  • 3.2 Stage 2: Learning weights over the reward features: Imagined trajectories are generated by uniformly sampling feature values within speculative minimum–maximum ranges produced in Stage 1, although some samples may be physically implausible.Preferences over these feature vectors can nevertheless support reward learning.
  • 3.2 Stage 2: Learning weights over the reward features: The method strategically selects candidate trajectory pairs for preference elicitation by maximizing uncertainty and restricting pairs to differ in at most two features.This restriction is intended to make salient differences easier for preference labelers to assess.
  • 3.2 Stage 2: Learning weights over the reward features: For stochastic preferences, uncertainty is estimated from disagreement among an ensemble of reward models, whereas noiseless preferences use disagreement among feasible reward functions.For noiseless preferences, the feasible-set range of signed return differences determines whether candidate reward functions disagree about the ordering.

4 Experiments

EARS is evaluated across three long-horizon environments using preference-based and natural-language inputs, with alignment measured by TAC without environment sampling. Across comparisons, EARS generally outperforms direct-prompting baselines and is competitive with methods trained on real trajectories, while feature-space sampling improves performance over observation-space sampling.

  • 4.1 Evaluating the learned reward function: EARS learns reward functions without sampling environment transitions and is evaluated across three environments, two ground-truth rewards per environment, and TAC alignment.TAC measures Kendall-Tau correlation between learned and ground-truth trajectory rankings using 500 evaluation trajectories.
  • 4.4 Results when specifying a reward function via ground-truth preferences: EARS learns more aligned reward functions than prompting baselines from ground-truth preferences, while privileged descriptions fail to produce alignment in all environments.EARS slightly underperforms direct prompting when all methods have higher TAC, but substantially outperforms otherwise.
  • 4.4 Results when specifying a reward function via ground-truth preferences: EARS is substantially more data-efficient than Offline-RLHF and tends to match or outperform Learning To Rank Real Trajectories despite using imagined trajectories.The comparison uses five seeds, with preferences labeled by the ground-truth reward function.
  • 4.4 Results when specifying a reward function via ground-truth preferences: Sampling imagined trajectories from the LLM-designed feature space substantially outperforms sampling them from the environment observation space.This tests an alternative EARS trajectory-sampling representation.
  • 4.5 Results when specifying a reward function via natural language: Natural-language evaluation compares realistic and privileged descriptions while keeping the remaining evaluation details consistent with the ground-truth-preference setting.The specifications are designed from the ground-truth reward function to emulate stakeholder-provided objectives.
  • 4.5 Results when specifying a reward function via natural language: With LLM-labeled preferences from natural-language specifications, EARS outperforms direct prompting in nearly all settings and performs especially better when baseline alignment is low.The exception is the traffic environment with the realistic reward description for r1, where EARS underperforms direct prompting.

5 Conclusion

EARS specifies reward functions from preferences without environment sampling and achieves stronger alignment than direct LLM reward generation across three decision-making domains and two preference-labeling settings.

  • Across three sequential decision-making domains, EARS produces reward functions more aligned with ground truth than direct LLM reward-generation alternatives.This holds for preferences generated by a ground-truth reward function and for LLM-labeled preferences conditioned on natural-language task specifications.
  • EARS matches or outperforms methods trained on preferences labeled over real trajectories.
  • Human-labeled preferences over imagined trajectories remain an open evaluation direction because stakeholders may not meaningfully reason about physically unrealizable trajectories.A domain-expert user study is proposed to address this question.

A Evaluating the learned reward function.

The evaluation uses TAC to compare learned and ground-truth reward rankings, while synthetic imagined-trajectory preferences are generated by mapping the ground-truth reward into the feature space.

  • TAC evaluates reward alignment by comparing how learned and ground-truth rewards rank trajectories, rather than measuring only empirical performance of trained policies.This choice avoids computationally expensive policy training and intractable optimal-policy derivation in long-horizon continuous environments.
  • Synthetic preferences over imagined trajectories are labeled using a mapped ground-truth reward function whose ranking matches the original transition-based reward.The mapped reward is scaled so the mean Bradley-Terry preference probability over evaluation trajectories is 0.9.

C Comparison to Sampling Imagined Trajectories from the Environment Observation Space

EARS samples imagined trajectories from a compact LLM-designed feature space, which substantially outperforms sampling from raw environment observations because irrelevant variables hinder informative exploration.

  • Sampling imagined trajectories from LLM-designed reward features substantially outperforms sampling from the environment observation space.The raw observation space contains redundant, objective-irrelevant features that attract uncertainty-based sampling toward uninformative trajectory differences.

D Implementation Details for LTRRT and Offline-RLHF

LTRRT and Offline-RLHF provide offline preference-learning comparisons using fixed pre-collected trajectories, while Table 4 contrasts designed-feature sampling with raw-observation sampling.

  • LTRRT and Offline-RLHF are trained and evaluated offline from the same fixed trajectory pool and differ only in trajectory representation.Their preference labeling, reward-model architecture, optimizer, and evaluation are otherwise identical.
  • Table 4 compares EARS using 2k stochastic preferences and LLM-designed features against a raw-observation ablation that bypasses Stage 1.

D.1 Preference-pair construction

Preference pairs are constructed from fixed pre-collected trajectories using two alternative feature representations: summed, normalized trajectory features or concatenated per-transition features.

  • 450 pre-collected trajectories per environment are uniformly paired from a fixed dataset for preference learning.The trajectories come from various policy-training checkpoints, and the pairing procedure uses 900 trajectory instances.
  • LTRRT (Learning To Rank Real Trajectories): LTRRT sums transition features into a d-dimensional trajectory vector and min–max normalizes each feature dimension.Its input dimension remains independent of trajectory length.
  • Offline-RLHF: Offline-RLHF concatenates all per-transition feature vectors, preserving the full feature trace but increasing input dimensionality to |τ|·d.Unlike LTRRT, these features are not normalized.

D.3 Training details

EARS training uses an LLM-driven, minimally tuned feature-design stage followed by structured imagined-trajectory sampling and either linear-program or ensemble-based preference learning.

  • Stage 1 uses a fixed-state stakeholder–facilitator dialogue to elicit, filter, and implement measurable reward features from the observation space.The dialogue produces executable Python features rather than relying on an open-ended turn budget.
  • The feature count d emerges from a stakeholder-specified small set of measurable objectives, with one-hot expansion for categorical objectives.EARS does not impose a target or maximum feature count.
  • Stage 1 generates speculative episode-level feature ranges, and Stage 2 samples each imagined feature independently within those ranges.Continuous features use uniform draws, discrete features use uniform choices, and categorical groups satisfy their ceiling constraint.
  • Candidate imagined pairs differ in at most two features to concentrate preference judgments on a small number of salient differences.This restriction is fixed a priori rather than tuned.
  • Noiseless preferences produce box-constrained, ℓ1-normalized weights via a feasibility linear program, while Bradley–Terry preferences use uncertainty from a 10-member ENN ensemble.The noiseless solver constrains each weight to [−1, 1]; the stochastic method quantifies uncertainty through predicted-preference variance.
  • Synthetic labels come from calibrated ground-truth rewards, whereas natural-language experiments use preferences labeled by Gemini.Calibration targets a mean Bradley–Terry preference probability of 0.9 over evaluation trajectories.

E.3 Compute and software environment

EARS was run on a two-node internal cluster and requires only lightweight local weight learning because it never samples the environment.

  • EARS experiments used two Ubuntu 24.04.4 nodes, each with dual Xeon E5-2680 v4 CPUs, about 250 GB RAM, and a GTX 1080 Ti GPU.The nodes provide 28 physical cores, 56 hardware threads, and 11 GB GPU memory in total per node.
  • Because EARS never samples the environment, its local computation is limited to lightweight Stage 2 reward-weight learning.The passage identifies feasibility linear-program computation as part of this local workload.

F Ground-truth reward function descriptions

The paper describes ground-truth rewards and realistic or privileged natural-language specifications across pandemic mitigation, glucose monitoring, and traffic control.

  • Ground-truth reward descriptions cover three domains: pandemic regulation, glucose monitoring, and traffic control.Both privileged and realistic descriptions are provided for the evaluated reward functions.
  • Privileged descriptions are hypothesized to be more specific than stakeholder-provided descriptions, whereas realistic descriptions are intended to reflect what a human could reasonably specify.The descriptions are supplied both to the direct-prompting baseline and to the LLM preference labeler.
  • Pandemic Mitigation: Pandemic objectives prioritize minimizing critical cases while favoring lower restrictions when safe; one description also emphasizes policy consistency.The ground-truth functions use regulation stage and critical-case quantities, with additional epidemiological state variables for r1.
  • Glucose Monitoring: Glucose objectives prioritize healthy blood-glucose levels and efficient insulin use, while an alternative description prioritizes minimizing treatment cost.The ground-truth descriptions define glucose readings, insulin doses, hypoglycemia, and patient-death termination.
  • Traffic Control: Traffic objectives emphasize fleet speed, efficient flow, throughput, safe following distances, and passenger comfort.The ground-truth traffic variables include vehicle speeds, target velocity, leader headway, accelerations, and a one-second minimum time gap.
Loading 2609.15544v1…