PaperlayerView exact revision on arXiv

Source-linked AI summary

STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

Yongqi Tong, Zhenyu Zhang, Ruirui Wang, Kewei Fu, Shaoqing Lin, Sijie Dong, Jiang-Ming Yang, Xin Zhang, Jianshe Li

arXiv:2608.16553v1cs.CL

TL;DR

Multi-preference RLHF leaves open when each objective should enter optimization. STAGE uses stability-gated cumulative admission with probing-derived ordering and adaptive weighting, and achieves higher automatic-evaluation averages than simultaneous and adapted sequential baselines in a 15-preference, 16-benchmark setting.

  • Problem

    Multi-objective RLHF lacks evidence on when to expose objectives during staged training and how to admit new dimensions without destabilizing earlier behavior.

  • Method

    STAGE cumulatively retains admitted objectives and admits one new dimension after reward-deviation gates or patience exhaustion, using probing-derived ordering and adaptive weighting.

  • Results

    STAGE achieves higher automatic-evaluation averages than simultaneous scalarization and shared-budget adapted baselines across the 15-preference, 16-benchmark evaluation.

  • Takeaways & Limitations

    Objective-entry timing is a concrete control variable for multi-preference RLHF, with ablations supporting retention, gated admission, and probing-derived ordering.

  • Takeaways & Limitations

    Validation uses automatic judge-based, single-run evaluations with two backbones and a fixed 15-objective inventory, without human preference or multi-seed validation.

Abstract

from arXiv · show

Multi-preference alignment is often framed as scalarization: combine reward dimensions, then optimize. This leaves a temporal decision underspecified: when should each preference dimension enter policy optimization? We propose \methodname, a stability-guided active-set controller for controlled objective admission. \methodname starts from a small active set, retains admitted objectives, and expands when reward-deviation gates indicate low recent deviation or a patience budget is exhausted. A probing phase estimates a hard-to-easy order, and adaptive weighting emphasizes underperforming active dimensions. Automatic evaluations with 15 training preferences and 16 held-out benchmark columns show that \methodname obtains higher averages than simultaneous scalarization and shared-budget adapted baselines. Component ablations and expansion dynamics further support cumulative retention, gated admission, and probing-derived ordering as useful design choices in this setting. These results position objective-entry timing as a concrete control variable in reward-vector RLHF.

1 Introduction

The section frames multi-preference RLHF as a temporal objective-admission problem: deciding when preference dimensions enter optimization, not only how active objectives are combined. It introduces STAGE, which probes dimensions, retains admitted objectives, adaptively weights them, and expands the active set under stability gates or patience exhaustion.

  • Motivation: Multi-objective RLHF must balance behaviors such as helpfulness, safety, factuality, refusal behavior, candor, reasoning quality, and creativity.Policy updates can improve one behavior while weakening others, motivating multi-objective training even with aggregate preference rewards.
  • Problem: The central operational question is when to expose a staged policy to additional objectives, because all-at-once training may amplify conflicts or let easy objectives dominate.The proposed framing treats objective-entry timing as distinct from reward representation, combination, or scheduling.
  • Method: STAGE is a stability-guided active-set controller that probes preference dimensions, starts with a small active set, retains admitted objectives, and expands after stability criteria or patience exhaustion.Its expansion uses pointwise and windowed reward-deviation gates, while probing estimates a hard-to-easy order.
  • Design: STAGE combines cumulative retention with gated admission, delaying new dimensions until recent active-set deviations are locally small while bounding each stage with a patience budget.Adaptive weighting emphasizes underperforming active dimensions, and the controller is tested across 15 preference dimensions with heterogeneous learning rates and cross-objective conflicts.
  • Evaluation: The evaluation uses Qwen3-0.6B and Llama3-8B-Instruct, with 15 training preferences, held-out automatic benchmarks, component ablations, active-set dynamics, and patience analysis.The reported analyses isolate adaptive weighting, reward-deviation gates, probing-derived ordering, and the efficiency-stability trade-off.

2 Related Work

Prior multi-objective alignment methods combine or weight already-selected reward dimensions, while STAGE controls when objectives enter a cumulative active set. It differs from pair- and sample-scheduling curricula by using probing and reward-deviation stability to govern objective admission.

  • Multi-Objective Alignment: Multi-objective alignment decomposes broad quality into sub-rewards, then optimizes dense decomposed rewards or combines reward dimensions during joint updates.These approaches address how to combine, project, or weight multiple reward dimensions after the active objectives have been chosen.
  • Multi-Objective Alignment: STAGE controls temporal active-set growth, whereas scalarization and Pareto rules control within-stage updates after objectives are selected.Its central distinction is asking when a new objective should enter the active set at all.
  • DPO-Style, Data-Centric, and Sequential Baselines: RCS schedules pair eligibility, Curri-DPO schedules sample pairs, and SPO schedules isolated objectives in a fixed sequence, whereas STAGE schedules a cumulative active objective set.STAGE retains previously introduced dimensions and conditions expansion on measured reward-deviation stability rather than fixed stage boundaries.
  • Curriculum Learning for Alignment: STAGE defines curriculum difficulty over preference dimensions, using probing to estimate hardness or volatility and reward-deviation gates plus a patience budget to decide advancement.Existing alignment curricula generally define difficulty over examples or response pairs.

3 Method: STAGE

STAGE controls when preference objectives enter PPO by cumulatively expanding an active set rather than optimizing all dimensions simultaneously. It estimates a hard-to-easy order through probing, advances when deviation-based stability criteria or a patience budget is met, and adaptively emphasizes weaker active dimensions.

  • Active-set control: STAGE admits preference dimensions cumulatively: inactive dimensions are excluded from the stage reward, while admitted dimensions remain active as the objective becomes progressively richer.At stage k, the active set is P_k = {π_1, ..., π_k}; each transition adds exactly one dimension and preserves earlier objectives.
  • Probing-derived ordering: A short joint probing phase ranks preferences by difficulty, producing a hard-to-easy expansion order used at the cluster level.The difficulty score combines low early gain with high volatility; observed hard preferences include creativity, reasoning quality, and numerical sensitivity.
  • Stability-gated advancement: Stage advancement requires joint instantaneous and windowed deviation stability across the active set, or occurs when the predefined patience budget is exhausted.The instantaneous gate rejects transitions after a single unstable batch, whereas the windowed gate captures accumulated short-window fluctuations; the gates assess local stability rather than mastery or Pareto optimality.
  • Adaptive weighting: Within each stage, adaptive weighting assigns larger weights to lower-scoring active preferences while maintaining normalized scalar reward aggregation.The implementation uses g(r) = 1−r to emphasize weaker dimensions during early stages; weighting is a within-stage module rather than STAGE’s main novelty.

4 Experiments

Experiments evaluate STAGE across two policy backbones and 16 held-out benchmark columns, finding higher averages than shared-budget baselines. Ablations, scorer audits, and expansion analyses support adaptive weighting, gated cumulative admission, probing-derived ordering, and moderate patience.

  • Evaluation setup: Evaluation uses 16 held-out benchmark columns across six alignment categories, distinct from the 15 training preference dimensions.Scores are oriented so higher is better, with generative outputs assessed using a single-response 1–10 quality/helpfulness prompt.
  • Main results: 44.81 average on Qwen3-0.6B and 62.93 on Llama3-8B-Instruct put STAGE 5.49 and 5.34 points above the strongest adapted baselines, respectively.On OrB, STAGE reaches 88.95/93.20 across the two backbones, and on OrB-h it reaches 63.92/65.80.
  • Component ablations: Adaptive Preference Weighting, stability gates, and probing-derived Preference Difficulty Ordering each contribute to the aggregate gain, with the latter raising the average from 42.22 to 44.81.APW emphasizes lower-scoring active dimensions, while ISG and WSG use reward-deviation gating for expansion.
  • Scorer sensitivity: STAGE remains top-ranked under the training scorer, whose agreement with the held-out judge is Pearson r=0.998 across cells and Pearson r=0.9998, Spearman ρ=0.976 for method averages.Table 4 repeats the primary Qwen3-0.6B comparison using Qwen3-235B-A22B-Instruct.
  • Expansion dynamics and patience: The development score rises as the active set grows toward all 15 dimensions, while patience affects the trade-off between rapid advancement and stabilization.Averages are 42.47 at T_max=15, 44.50 at T_max=30, and 45.23 at T_max=100; the headline configuration uses T_max=100.

5 Conclusion and Future Work

STAGE frames objective-entry timing as a control problem in multi-preference alignment, using stability-guided active-set expansion and supporting diagnostics. In a 15-dimensional setting, it achieves higher average automatic-evaluation scores than specified baselines, while future work includes GRPO and offline variants.

  • Core contribution: STAGE is a stability-guided active-set controller that controls when preference objectives become active, not only how they are combined.Its mechanisms include cumulative expansion, probing-derived ordering, adaptive weighting, reward-deviation gates, and a patience fallback.
  • Results: In a 15-dimensional setting, automatic evaluations show higher averages for STAGE than for simultaneous scalarization and shared-budget adapted sequential baselines.The conclusion reports higher averages without specifying numerical values.
  • Design validation: Ablations and diagnostics support retaining prior objectives, gating new admissions with reward deviation, and using probing to order expansion.These findings support cumulative retention, gated admission, and probing-derived ordering as controller design choices.
  • Implications and future work: The results position objective-entry timing as a concrete lever for multi-preference RLHF, with future work targeting GRPO and offline variants.The supplied passage also mentions dynamic work, but the sentence is truncated after “dynamic.”

Limitations

The validation is limited to automatic judge-based, single-run evaluation with a fixed 15-objective inventory, leaving human preference validation, multi-seed uncertainty, benchmark-native checks, and scaling behavior unmeasured. Future work should broaden the empirical setting and extend STAGE to other optimization methods and dynamic objective ordering.

  • Empirical limitations: Validation remains limited to automatic judge-based evaluation and single-run estimates across two policy backbones and 16 held-out benchmark columns.The study does not yet include human preference validation, multi-seed uncertainty estimates, or benchmark-native checks.
  • Empirical limitations: The setup fixes a 15-objective inventory and does not measure scaling behavior over other objective counts or compositions.Controlled sweeps over the number and composition of objectives are identified as future work.
  • Future work: Future evaluation should use larger and more varied policy backbones, multi-seed estimates, and human-preference validation.These directions are proposed to broaden the empirical setting beyond the current validation design.
  • Future work: STAGE could be extended beyond PPO-style optimization to GRPO or offline preference optimization.The proposed methodological extensions target optimization settings not covered by the current setup.
  • Future work: Fixed probing-derived ordering could be replaced with dynamic reordering when objective difficulty shifts during training.This would address changes in relative objective difficulty over the course of training.

A Evaluation Benchmarks

The held-out evaluation follows recommended protocols and organizes benchmarks into six categories. It covers mis/disinformation, toxicity and spam, sensitivity, helpfulness under safe-but-challenging prompts, and faithfulness using named datasets and task-specific scoring.

  • The held-out evaluation is organized into six categories following Chen et al. (2022)'s recommended protocols and dataset selections.
  • Mis/disinformation uses CG, LUN, and Sat., while toxicity and spam use HSOL, Jigsaw, and toxic OrBench.
  • Sensitivity is evaluated with EDENCE and FAS, covering privacy and sensitive topics.
  • Helpfulness under safe-but-challenging prompts uses OrB-h and Mor.; scores are transformed as 100 × (1 −score) so higher values indicate better helpfulness/safety trade-offs.
  • Faithfulness is measured with TruthfulQA-MC1 (TQA).

B Data Sources and Construction Details · B.1 Source Aggregation

The PPO training set aggregates 10 data sources to promote task diversity and preference coverage across multiple domains.

  • B.1 Source Aggregation: 10 data sources are aggregated into the PPO training set.The aggregation is presented as the basis for the training-set construction.
  • B.1 Source Aggregation: The source aggregation is designed to encourage task diversity.Task diversity is one stated motivation for combining the sources.
  • B.1 Source Aggregation: The aggregation is also designed to encourage preference coverage.Preference coverage is identified alongside task diversity as a construction goal.
  • B.1 Source Aggregation: The PPO training data spans multiple domains through this source aggregation.The passage characterizes the intended coverage as cross-domain.
  • B.1 Source Aggregation: Source aggregation links task diversity with preference coverage in the PPO training set.Both objectives are stated as reasons for aggregating the data sources.
  • B.1 Source Aggregation: The construction details identify source aggregation as a deliberate design choice for PPO training.The passage introduces the 10-source aggregation as part of the training-data construction.

B.2 Construction Methodology

The dataset construction pipeline identifies which of 15 preference dimensions are relevant and discriminative for each query, then uses stratified sampling to balance preference coverage in a final 20k-query dataset.

  • Preference Discrimination Tagging: A fixed automatic scorer returns a binary mask over 15 rubric items, tagging each query’s relevant and discriminative preference dimensions.The scorer receives the query and the full list of rubric items.
  • Stratified Sampling: Stratified sampling constructs the final 20k-query dataset by equalizing selected-query counts across preferences marked discriminative, mitigating dominance by common intents.The method targets balanced representation of each preference dimension during PPO training, subject to tagging accuracy.

C Training Details · D Baseline Details · D.1 Reward Consistency Sampling (RCS-adapted)

The paper specifies PPO training configurations in veRL and adapts RCS through cross-model preference construction, preserving a shared 20k-instance budget across 15 training dimensions. The RCS adaptation retains Pareto-consistent preference logic while using teacher responses as chosen examples and base-model responses as rejected examples.

  • C Training Details: Main experiments use the veRL framework with PPO for all models.Training configurations cover trainer, reinforcement-learning algorithm, model-specific settings, and probing budgets.
  • C Training Details: Configured parameters and probing budgets are summarized in Table 6.The table provides the detailed settings used for the trainer, reinforcement-learning algorithm, and models.
  • D.1 Reward Consistency Sampling (RCS-adapted): RCS filters preference pairs whose reward signals conflict.It retains pairs where the winning response scores at least as highly as the losing response across every objective.
  • D.1 Reward Consistency Sampling (RCS-adapted): The RCS-adapted implementation modifies data construction to equalize training scale across baselines.This adaptation preserves the comparison’s shared data-volume constraint while retaining RCS’s consistency criterion.
  • D.1 Reward Consistency Sampling (RCS-adapted): The adaptation uses Qwen3-235B-A22B-Instruct responses as Chosen examples and Qwen3-0.6B responses as Rejected examples.This cross-model pairing replaces the original construction strategy for the reported replication.
  • D.1 Reward Consistency Sampling (RCS-adapted): The distillation-based adaptation preserves 20k training instances with usable margins across 15 training dimensions.Detailed dimension rubrics are provided in Appendix J.

D.2 Curriculum-DPO (Curri-DPO) … E Prompt Index

The paper implements curriculum-based, sequential, and scalarized PPO baselines under a shared comparison framework. Curri-DPO uses staged response difficulty, SPO-adapted trains preferences sequentially, and PPO baselines compress preference signals into scalar rewards.

  • D.2 Curriculum-DPO (Curri-DPO): Curri-DPO-adapted trains through three sequential stages, pairing target responses with rejected responses from increasingly larger models.The stages use Qwen3-0.6B, Qwen3-8B, and Qwen3-30B as rejected-response sources.
  • D.2 Curriculum-DPO (Curri-DPO): The curriculum begins with pairs having the largest model-scale gaps and advances toward pairs with smaller differences.This adapts Curri-DPO’s easy-to-hard difficulty schedule to the response pool and shared budget.
  • D.3 Sequential Preference Optimization (SPO-adapted): SPO-adapted trains the 15 preference dimensions in 15 sequential stages, while SPO-Reverse uses the opposite hard-to-easy order.SPO uses the preceding model as a reference anchor and applies a constrained loss at each stage.
  • D.4 PPO: PPO baselines represent two primary reward-scalarization paradigms and use the same 20k-query pool and 600-step PPO budget as STAGE.The common budget keeps comparisons centered on scheduling strategies, while reward-query formats vary by baseline objective.
  • D.4 PPO: Vanilla Multi-objective PPO prompts the scorer to synthesize all 15 criteria into one unified quality score from 1–5.This tests credit assignment when potentially conflicting preference signals are compressed into a single scalar.
  • D.4 PPO: Reward Soups converts a 15-dimensional criterion-score vector into a scalar using fixed weights w_i = 1/15 before advantage estimation.This is the simultaneous-optimization baseline.
  • E Prompt Index: The prompt index lists materials for preference-label mapping, training-dataset construction, PPO baselines, RCS and Curri-DPO rubrics, and task-specific scoring.The listed scoring categories cover generative tasks and binary classification.

F Preference Label Mapping … M Use of LLMs

The appendix specifies canonical preference-key mappings, evaluation and training prompts, task-specific scoring formats, and the authors’ use of LLMs. Together, these materials define reproducible interfaces for preference assessment and clarify that writing assistance did not determine experimental results or methods.

  • F Preference Label Mapping: Table 7 maps manuscript preference names to compact implementation keys used in prompts, JSON outputs, and reward-vector implementations.The mapping is described as canonical and includes names such as Ethical compliance → ethical and Reasoning quality → reasoning.
  • G Prompt: Training Dataset Construction: The training-data prompt asks whether each question differentiates chatbot behavior across 15 canonical preferences and requires JSON lists of differentiable and non-differentiable preferences.The prompt requires canonical snake_case keys in the JSON arrays.
  • H Prompt: Vanilla Multi-objective PPO: Vanilla multi-objective PPO uses one overall quality score on a 1-to-5 scale across 15 criteria and returns only a JSON object with score and comment fields.The criteria include ethical, accuracy, helpfulness, reasoning, creativity, operational, and question answering dimensions.
  • I Prompt: Reward Soups: Reward Soups scores all 15 criteria separately on a five-point scale, with snake_case score keys and corresponding brief comments in JSON.The prompt explicitly requires a separate score and justification for each criterion.
  • J Prompting: RCS and Curri-DPO Rubric: The RCS and Curri-DPO rubric defines five-point behavioral criteria for 15 preferences and requires paired snake_case scores and comments in a single JSON object.Examples specify standards for ethical behavior, accuracy, candor, clarification, and other preference dimensions.
  • K Prompt: Scoring for Generative Tasks: Generative-task evaluation asks for a single quality-and-helpfulness score from 1 to 10 with no additional text or explanation.The required response is only one number between 1 and 10.
  • L Prompt: Scoring for Binary Classification Tasks: Binary-classification evaluation checks semantic consistency with the ground truth and requires only 1 for correct or 0 for incorrect.The prompt supplies text, prediction, and label fields before requesting a single binary response.
  • M Use of LLMs: General-purpose LLMs supported wording, organization, consistency, and formatting, while frozen LLMs served as reward and evaluation models; writing assistance did not generate results or choose methods.The disclosure separates manuscript preparation from the experimental use of frozen LLMs.
Loading 2608.16553v1…