PaperlayerView exact revision on arXiv

Source-linked AI summary

On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning

Baptiste Bonin, Caro Strickland, Audrey Durand

arXiv:2609.26918v1cs.LGcs.AI

TL;DR

Preference relabeling extends hindsight experience replay to preference-conditioned MORL, but whether an achieved trade-off is a valid substitute for a requested preference has not been directly tested. The paper evaluates the practice, identifies Preference Coverage Collapse, and introduces mixed relabeling; it reports that the practice is harmful in most affected settings and that mixing repairs most of those harms.

  • Problem

    Whether HER-style substitution of achieved preferences is valid in MORL had not been directly tested, despite its adoption from goal-conditioned RL.

    Source: §1 · p. 2

  • Method

    The paper evaluates preference relabeling across off-policy MORL algorithms, diagnoses preference-space coverage changes, and proposes her_mix, which blends achieved and requested preferences.

    Source: §1 · p. 2, §2.6 · p. 3, §4.1 · p. 4

  • Results

    19 of 36 settings degrade by as much as four standard deviations, exactly one improves, and her_mix recovers 16 of the 19 harmed settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Achieved-preference relabeling should not be assumed safe by analogy to HER; the paper concludes that protecting preference-space coverage makes it safer within the studied settings.

    Source: §7 · p. 7

    Evaluation is limited to MO-MuJoCo with two or three objectives and four algorithm/sampler combinations over two critic backbones; discrete-action, higher-objective, multi-policy, and model-based settings remain untested.

    Source: §7 · p. 7

Abstract

from arXiv · show

Hindsight relabeling which retroactively replacing a transition's goal with the outcome the agent actually achieved is an effective tool for improving sample-efficiency in Reinforcement Learning (RL). A natural extension to preference-conditioned multi-objective RL (MORL) relabels transitions with the preference direction the agent achieved rather than the one asked for. We show that this extension is frequently harmful: across four preference-conditioned off-policy algorithms spanning two critic backbones and two preference-sampling schemes on the continuous-control MO-Gymnasium suite, it degrades 19 of 36 algorithm-environment settings by as much as four standard deviations, improves only one, and leaves the rest unaffected. The harm is not a symptom of noisy relabels; denoising the target recovers almost nothing, and neither prioritized sampling nor any buffer-structural choice reproduces it. Instead, repeated relabeling collapses the critic's coverage onto whatever narrow region of the preference space the agent happened to visit. We name this failure mode \emph{Preference Coverage Collapse}, and quantify it with abandoned preference mass (APM), a value-aware statistic that tracks the harm ($\rho = -0.73$) where a purely structural coverage count does not. We then introduce \texttt{her\_mix}, a single-parameter convex combination pulling the achieved direction back towards the requested preference. At one fixed value across every algorithm and environment, it returns 16 of the 19 harmed settings to baseline, preserves and even improves the one setting in which relabeling helps, and cuts abandoned preference mass from $69\%$ to $6\%$. Protecting coverage over the preference simplex, not filtering noisy relabels, is what makes hindsight relabeling safe for MORL.

1 Introduction

Preference relabeling appears to offer free data reuse in MORL, but unlike goals, achieved trade-offs are not necessarily valid substitutes for requested preferences. The paper tests this practice and finds it harmful in most settings where it has an effect.

1 Introduction

  • MORL agents must learn an entire trade-off space from a finite experience pool, compounding the data demands of reinforcement learning.HER turns failed goal-conditioned trajectories into learning signal by substituting achieved outcomes for assigned goals.Source: §1 · p. 1
  • An achieved trade-off may say little about the requested preference, so repeated relabeling can train the critic on a narrow corner of preference space.The paper names this failure mode Preference Coverage Collapse.Source: §1 · p. 1
  • Whether achieved-preference substitution remains well-defined in MORL had not been directly tested, despite its adoption by prior work.The paper asks whether it improves off-policy MORL, where it fails, why, and whether it can be made safe.Source: §1 · pp. 1–2
  • Relabeling degrades 19 of 36 algorithm-environment settings by as much as four standard deviations and improves exactly one.The controlled evaluation spans four preference-conditioned algorithms, two critic backbones, and two preference-sampling schemes.Source: §1 · p. 2

2 Background

The background introduces preference-conditioned MORL and the replay-based learning setting, then contrasts HER's goal substitution with preference relabeling, whose validity does not automatically transfer.

2.2 Multi-Objective Reinforcement Learning

  • In MORL, vector-valued rewards replace scalar rewards, and a preference-conditioned policy adapts one policy to a preference vector at inference time.The paper focuses on single-policy preference-conditioned methods.Source: §2.2 · p. 2

2.3 Evaluation Metrics

  • Off-policy actor-critic methods repeatedly sample stored transitions, while prioritized experience replay changes sampling frequency without changing transition content.The paper evaluates Pareto-front recovery with hypervolume and expected utility; hypervolume depends on a reference point, while expected utility does not.Source: §2.3 · p. 2, §2.4 · p. 3

2.5 Hindsight Experience Replay

  • HER adds transitions relabeled with an achieved future goal because goal proximity defines success for whichever goal is supplied.That makes swapping the assigned goal for an achieved state preserve the reward definition.Source: §2.5 · p. 3

2.6 Preference Relabeling in MORL

  • Preference relabeling maps an achieved objective vector to the preference simplex by clipping negative components and normalizing the result.The relabeled transition is stored alongside the original.Source: §2.6 · p. 3
  • Unlike HER's sparse-reward setting, MORL scalarized rewards are generally nonzero, so preference relabeling is motivated by data reuse rather than signal recovery.An achieved trade-off is not a preference the agent necessarily aimed for or could realize by optimizing for it from the start.Source: §2.6 · p. 3
  • Clipping makes the default operator ill-posed for objectives that are negative by construction, since those objectives can never receive relabeled preference weight.The paper evaluates a sign-robust alternative based on per-objective min-max normalization.Source: §2.6 · p. 3

3 Related Work

Related work traces HER relabeling choices, its adoption in MORL, and coverage research, distinguishing this paper's diagnostic use of diversity from approaches that optimize or structurally protect coverage.

3.1 HER in single-objective RL

  • HER's relabeling target matters: prior work favored near-future states, and poor performance with distant goals motivated intermediate-goal curricula.The paper notes that this scrutiny of target choice has not carried over to MORL's adoption of the substitution.Source: §3.1 · p. 3

3.2 HER-style relabeling applied to MORL

  • Prior MORL methods adopted HER-inspired replay or achieved-trade-off substitution by analogy, without analyzing whether achieved outcomes are valid preference substitutes.PD-MORL samples preferences independently of the trajectory, whereas HPR implements achieved-trade-off substitution directly.Source: §3.2 · p. 3

3.3 Off-policy MORL algorithms

  • Off-policy MORL varies both critic/update rules and preference-sampling schemes, including uniform simplex sampling and sampling focused on informative regions.The cited approaches include scalarized DQN-style learning, continuous-control backbones, cone-based sampling, and generalized policy improvement.Source: §3.3 · pp. 3–4

3.4 Diversity metrics for coverage

  • Coverage research includes entropy-based effective-category measures and architectures that improve or regularize preference-space coverage.This paper instead uses a diversity metric diagnostically to characterize coverage changes caused by relabeling, not to optimize diversity.Source: §3.4 · p. 4

4 Preference Coverage Collapse

The paper characterizes preference coverage collapse with structural and value-aware measures, then proposes mixed relabeling to preserve the requested preference's influence.

4.1 Coverage

  • APM measures the fraction of preferences whose utility falls meaningfully below a baseline, while effective coverage counts how many distinct trade-offs a policy serves.APM uses a 10% threshold here; the paper cautions that APM recasts harm as lost coverage rather than independently explaining its cause.Source: §4.1 · p. 4

4.2 Mixed Relabeling

  • her_mix forms each relabeled preference as a convex combination of the collected and achieved preferences, using λ = 0.25 throughout without per-setting tuning.The mixing keeps the result within the preference simplex; a TD-gated variant is proposed as a control against the alternative noisy-outlier explanation.Source: §4.2 · p. 4

5 Experiments

The experiments compare four off-policy MORL algorithms across continuous-control tasks, varying only the replay buffer. Paired evaluations measure final EUM and preference-space coverage under a common protocol.

5.1 Algorithms and environments

  • Four algorithms form a 2 × 2 grid of critic update rules and preference samplers, separating effects associated with either design axis.CAPQL and GPI-PD differ on both axes; MO-TD3 and CAPQL-uniform swap one component each.Source: §5.1 · p. 5
  • The benchmark crosses four algorithms with nine continuous-action MO-Gymnasium tasks, yielding 36 algorithm-environment settings.The suite includes eight MO-MuJoCo tasks and mountaincar-continuous; 1mo-reacher-v5 is excluded for its discrete action space.Source: §5.1 · p. 5, §5.3 · p. 5

5.2 Protocol; 5.3 Statistical methodology

  • Runs train for 150k steps and use paired seeds; final EUM is primary, while coverage metrics distinguish diversity that scalarized scores may miss.Claims use 16 seeds per setting (32 on CAPQL hopper tasks); mechanism probes use 3 seeds. HV is secondary, and no-effect claims require equivalence testing.Source: §5.2 · p. 5, §5.3 · p. 5

6 Results

The results establish that preference relabeling can harm learning, identify preference coverage collapse as a central failure pattern, and test mixed relabeling as a repair. The study also checks that normalization artifacts and alternative buffer explanations do not account for the reported effects.

6.1 The Standard Operator Silently Ignores Cost Objectives

  • On cost-objective tasks, 86–100% of standard relabels degenerate to a corner or uniform vector; sign-robust normalization changes ant-2obj from d = −1.41 to +0.04.The harm remains on hopper-2obj, where clipping does not bind; subsequent results use the sign-robust operator.Source: §6.1 · p. 5

6.2 Well-Posed Relabeling Degrades Learning

  • Across 36 settings, relabeling harms 19, leaves 16 indistinguishable from baseline, and improves one, with effects from d = −4.03 to +1.03.Harm is algorithm- and environment-dependent: three-objective hopper degrades under all four algorithms, while humanoid is unaffected throughout.Source: §6.2 · p. 5, Figure 2 · p. 6

6.3 The Harm is intrinsic to Relabeling

  • Prioritized sampling and buffer structure do not explain the harm, and return-to-go denoising recovers only 7% of the loss.Swimmer and halfcheetah never terminate early yet are among the most harmed (d = −2.87, −4.03).Source: §6.3 · pp. 5–6

6.4 The Harm Is Preference Coverage Collapse

  • Relabeling abandons a mean 69% of preference mass on harmed settings, and abandoned preference mass tracks harm more strongly than effective coverage C (ρ = −0.73 versus +0.40).On harmed settings, mixing cuts APM from 69% to 6% and restores 87% of lost C; preference-wise utility remains near the achieved direction but falls elsewhere.Source: §6.4 · p. 6, Figure 3 · p. 6
  • On hopper-2obj, harm appears only near total substitution: d shifts from +0.15 at f = 0.25 and +0.02 at f = 0.5 to −3.48 at f = 1.Relabeled TD error is smaller than collected-conditioning TD error in 34 of 36 settings, opposing a noisy-target explanation.Source: §6.4 · p. 6, Figure 4 · p. 7

6.5 Mixed Relabeling Restores Coverage

  • Mixed relabeling recovers 87–117% of harm on five pre-registered harmed settings, while the TD gate recovers only 62% on halfcheetah and swimmer and fails both criteria.Mixing is not simple attenuation: recovery is 94–103% for λ ≤ 0.5 on hopper-2obj, and the blend outperforms both endpoints on ant.Source: §6.5 · pp. 6–7, Table 1 · p. 7, Figure 4 · p. 7

6.6 Generalisation across the suite

  • At fixed λ = 0.25, mixed relabeling recovers harm on 16 of 19 harmed settings and leaves 15 of 16 neutral settings unaffected across the full suite.The helped setting retains its benefit; the one neutral-setting regression is GPI-PD mountaincar (d = −0.66, CI [−1.07, −0.36]).Source: Figure 2 · p. 6, §6.6 · p. 7

7 Discussion, Limitations, and Future Work

The discussion presents her_mix as a repair for preference coverage collapse while emphasizing that the findings have bounded scope. The appendix details the algorithms and preference-sampling design used in the study.

7 Discussion, Limitations, and Future Work

  • her_mix removes relabeling harm without per-environment tuning and can outperform both pure relabeling and baseline by reaching trade-offs neither endpoint trains.Effective coverage C is a weaker harm indicator than APM, which tracks harm and reads near-zero on the setting where relabeling helps.Source: §7 · p. 7
  • The evidence is limited to MO-MuJoCo tasks with two or three objectives and four algorithm/sampler combinations; broader MORL settings remain untested.At λ = 0.25, her_mix recovers or preserves 32 of 36 verdicts, with one regression and partial recovery on the largest original harm.Source: §7 · p. 7

A Algorithm Details

  • The four algorithms vary critic update rule and preference sampling, while library-default hyperparameters are used and the replay buffer is the only varied component.CAPQL uses a SAC-based entropy-augmented critic and cone sampling; GPI-PD uses TD3 and a support set, while MO-TD3 and CAPQL-uniform complete the update-rule-by-sampler grid.Source: A Algorithm Details · p. 9

B The Clip Artifact on Cost Objectives

The standard clip-based achieved-preference operator can erase negative cost objectives, producing degenerate relabels. Per-objective min-max normalization removes this artifact where clipping binds, but does not eliminate the broader harm.

B The Clip Artifact on Cost Objectives

  • On four tasks with negative control or energy costs, clipping achieved objectives at zero prevents relabeled preferences from pointing toward the cost objective.The resulting preferences degenerate to a corner or uniform vector.Source: B The Clip Artifact on Cost Objectives · p. 9
  • Replacing clipping with per-objective min-max normalization makes ant-2obj statistically equivalent to baseline and shrinks the effect by more than half on swimmer and halfcheetah.Walker2d remains substantially harmed, while hopper-2obj is unchanged because clipping never binds there; main-text results therefore use the sign-robust operator.Source: B The Clip Artifact on Cost Objectives · p. 9, Table 2 · p. 9

C Alternative Explanations for the Harm

C.1 Prioritized sampling

Prioritized sampling does not consistently explain relabeling harm, while early termination amplifies but does not create it. The clip-based operator also makes the termination interaction unidentifiable.

C.1 Prioritized sampling

  • On six of eight environments the PER × HER interaction interval crosses zero, and the two identifiable environments disagree in sign.PER alone ranges from d = +0.05 to −0.43, so it is neither an independent source of harm nor a consistent amplifier.Source: §C.1 · p. 9, Table 3 · p. 9

C.2 Early termination

  • Early termination amplifies relabeling harm, but does not create it: swimmer and halfcheetah never terminate early and remain among the most harmed settings.Under the clip-based operator, the termination interaction is not identifiable (J = −14.4 [−59.1, +34.9], p = 0.54).Source: §C.2 · p. 10

C.3 Relabel-direction noise and episode length

Changing the relabel direction to return-to-go recovers little of the loss, while suppressing relabels from short episodes recovers more. Neither intervention closes the gap, consistent with the harm arising from coverage rather than target noise or episode selection alone.

C.3 Relabel-direction noise and episode length

  • Return-to-go relabeling recovers 7% of the loss, while suppressing relabels from episodes shorter than 200 steps recovers 61% with relabeling still active on 76% of transitions.Short failed episodes are disproportionately responsible, but no filter closes the gap and denoising the target barely moves it.Source: §C.3 · p. 10

D Structural Buffer Design at Conclusion-Grade Power

At higher statistical power, none of the tested structural buffer choices harmed performance or approached relabeling’s effect size. The lone significant benefit did not replicate across environments.

D Structural Buffer Design at Conclusion-Grade Power

  • The re-test covered hopper-2obj and swimmer; buffer capacities at 1× and 4× training budget were omitted because the buffer never fills at 150k steps.The initial four-seed screen had minimum detectable effect |d| ≈ 2.38; the 16-seed re-run reduced it to |d| ≈ 1.02 and used TOST with δd = 0.5.Source: D Structural Buffer Design at… · p. 10
  • No tested structural choice harmed performance, and none approached relabeling’s d = −4.03; the sole significant benefit reversed sign across environments.Per-objective partitioning helped on hopper-2obj (d = +0.67, p = 0.022) but had d = −0.34 on swimmer.Source: D Structural Buffer Design at… · p. 10, Table 5 · p. 10
  • At 16 seeds per cell, six of sixteen cells were equivalent to baseline at δd = 0.5, while nine remained inconclusive at that margin.All inconclusive intervals were well inside ±1, much smaller than relabeling’s d = −4.03.Source: D Structural Buffer Design at… · p. 10

E Per-Setting Validation of the Fixed Coefficient

At a single fixed coefficient, her_mix recovers most harmed settings while retaining the relabeling benefit and leaving most neutral settings unaffected. The remaining harmed-setting exceptions have small original effects, making their recovery ratios noisy.

E Per-Setting Validation of the Fixed Coefficient

  • The fixed λ = 0.25 was assessed across 36 settings using pre-registered criteria for recovery, retaining a helped effect, or avoiding significant harm.For harmed settings, the criterion was recovery ≥70% or TOST equivalence to baseline; for helped settings, d ≥ +0.5 retained; for neutral settings, no significant harm.Source: E Per-Setting Validation of the Fixed… · p. 10
  • The three unrecovered harmed settings had small relabeling effects (|d| ≈ 0.6), while mixing’s absolute effect was already inside the equivalence band.Their recovery ratios are therefore noisy.Source: E Per-Setting Validation of the Fixed… · p. 10
  • In per-setting validation at fixed λ = 0.25, her_mix recovered 16 of 19 harmed settings; the single helped setting kept its benefit, and 15 of 16 neutral settings were unaffected.Source: E Per-Setting Validation of the Fixed… · p. 10, Table 6 · p. 11
Loading 2609.26918v1…