PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training

Junwon Ko, Dong-Jae Lee, Minchan Kwon, Sunghyun Baek, Junmo Kim

arXiv:2609.10052v1cs.CLcs.AIcs.LG

TL;DR

Trajectory-level outcome labels give LLM agents little supervision for preserving multiple successful branches from a shared decision state. DDO addresses this with state-aligned branch collection and reference-relative target-odds training, achieving the strongest joint task success and successful strategy coverage across BabyAI, BabaIsAI, and WebShop, plus the highest recovery rate after local action replacement.

  • Problem

    Trajectory-level outcome labels provide little supervision for preserving multiple successful branches from the same decision state.

  • Method

    DDO combines DTC, which constructs state-aligned branch sets, with RTO, which trains on reference-relative targets over successful branches.

  • Results

    DDO achieves the strongest joint task success and successful strategy coverage across BabyAI, BabaIsAI, and WebShop, and the highest recovery rate after local action replacement.

  • Takeaways & Limitations

    DDO broadens successful strategy coverage while maintaining higher task success than successful-only imitation and decoding-time diversification controls.

  • Takeaways & Limitations

    The experiments are confined to environments supporting state reconstruction and alternative branches from a shared decision state under a common success predicate.

Abstract

from arXiv · show

LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget. We present Direct Diversity Optimization (DDO), an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO). DTC constructs state-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference-relative targets over successful alternatives. DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful-only imitation and decoding-time diversification controls.

1 Introduction

Trajectory-level post-training can collapse an agent onto one successful path, leaving alternative successful branches underrepresented. DDO addresses this by creating same-state branch supervision and achieves the strongest joint task success and successful strategy coverage across three benchmarks.

  • In the shopping example, Model A follows one brand-page trajectory, while Model B retains distinct search, filtering, and comparison strategies.
  • When the brand-page trajectory is blocked, Model A has no fallback, whereas Model B retains an alternative path to success.
  • Trajectory-level outcome labels provide little supervision for preserving multiple successful branches from the same decision state.
  • DDO combines state-aligned branch construction with reference-relative optimization to convert trajectory-level outcomes into supervision over multiple successful alternatives.
  • DDO achieves the strongest joint task success and successful strategy coverage among compared post-training methods across BabyAI, BabaIsAI, and WebShop.

2 Related Work

Preference post-training can narrow output diversity, motivating methods that preserve multiple acceptable responses. DDO retains multiple successful branches as valid positives while distinguishing them from failed branches without imposing an arbitrary ranking among successes.

  • RLHF and DPO have been reported to reduce output diversity relative to the base model, while KL-regularized preference learning can amplify dominant preferences.
  • Sequential-decision methods use branching rollouts, trajectory-level contrasts, or step-level feedback to provide richer preference supervision.
  • DDO preserves multiple successful branches as valid positives, distinguishes them from failed branches, and avoids forcing an arbitrary ranking among successful branches.

3 Method

DDO uses Divergence-Tree Collection to build labeled branch sets rooted at shared states, then applies Reference-Relative Target-Odds training to compare successful alternatives and successful–failed pairs. Its targets preserve adjustable reference-relative weighting among successful branches while ordering successes above failures.

  • 3 Method: DDO converts trajectory-level outcome supervision into same-state branch comparisons and a target distribution over successful alternatives.
  • 3.1 Problem Setting and Notations: The problem setting supplies only binary termination outcomes for trajectories, with no per-step rewards or learned reward model assumed.
  • 3.2 Divergence-Tree Collection: DTC restores selected interior decision states, queries an expert for action-divergent alternatives, and labels each retained branch by rollout success or failure.
  • 3.3 Reference-Relative Target-Odds Objective: RTO aligns the model’s successful-branch log-odds with target odds from a reference-relative distribution, while success–failure pairs place successful branches above failed ones.
  • 3.3 Reference-Relative Target-Odds Objective: Smaller α flattens reference weights among successful branches, α = 0 makes the target uniform, and α = 1 preserves the reference distribution.

4 Experimental Setup

The evaluation spans navigation, rule manipulation, and web interaction, comparing DDO with preference-based and diversity-aware post-training baselines under shared protocols. Models are assessed by task success and successful strategy coverage under fixed rollout budgets.

  • Benchmarks: The benchmarks cover controlled navigation, rule manipulation, and web-style shopping behavior.BabyAI and BabaIsAI use BALROG text-action implementations with standardized observations and executable action interfaces.
  • Baselines: DDO is compared with DPO, frequency- and probability-based diversity baselines, and tie-aware DPO variants.DivFreq and DivProb use state-aligned branch data, while TieDPO-RK and TieDPO-Dav use Rao-Kupper and Davidson tie-aware objectives.
  • Comparison protocol: All post-training methods within each benchmark use the same DTC-collected branch sets, isolating objective differences while holding data fixed.A separate ablation varies the data side by replacing DTC with comparisons formed without DTC.
  • Metrics: Evaluation measures task success and successful strategy coverage under a fixed rollout budget.WebShop treats graded purchase scores of at least 0.9 as successful, while BabyAI and BabaIsAI require benchmark progression to reach 1.0.
  • Metrics: ESD counts distinct successful trajectory classes, while H-ESD decreases when successful rollouts concentrate on repeated classes.Normalizing by valid rollouts makes both metrics reflect finite-budget coverage that combines solving tasks with producing distinct successful strategies.

5 Results

DTC improves branch collection and downstream post-training, while DDO combines DTC and RTO for the strongest joint success and coverage across the evaluated benchmarks. The results isolate complementary contributions from broader branch supervision and the reference-relative objective.

  • 5.1 DTC Improves Branch Supervision: DTC discovers more unique successful trajectories than independent sampling under the same request budget on BabyAI and BabaIsAI.This increases collection-time successful strategy coverage before post-training begins.
  • 5.1 DTC Improves Branch Supervision: 56.4% of DTC pairs form at or beyond the trajectory midpoint, whereas 79.3% of pairs without DTC occur within the first 20%.DTC therefore supplies branch-level supervision across early, middle, and late decision depths.
  • 5.1 DTC Improves Branch Supervision: 10.5 pp, 0.08, and 0.09 are the average DDO gains from adding DTC for task success, H-ESD, and ESD across BabyAI and BabaIsAI.On BabyAI, H-ESD and ESD each increase by 0.06; on BabaIsAI, they increase by 0.10 and 0.12, respectively.
  • 5.2 Main Results: DDO achieves the highest benchmark-average success (0.76), H-ESD (0.35), and ESD (0.40).Relative to DPO, these correspond to gains of 8 pp, 0.09, and 0.09; WebShop shows a 10 pp success gain from 0.26 to 0.36.
  • 5.2 Main Results: DDO raises task success by 10 pp and both coverage metrics by 0.04 over the shared Reference initialization.Other post-training methods remain below Reference on both coverage metrics.
  • 5.2 Main Results: Across compared methods, DDO leads every benchmark-level aggregate and all three overall averages.DivFreq and DivProb change average success and coverage by at most ±0.02, while tie-aware DPO variants have lower average success than DPO.

6 Analysis

The analyses test whether DDO’s broader successful-strategy coverage improves recovery after disruptions and remains advantageous over imitation and decoding-time diversification. DDO achieves stronger recovery and preserves a better success–coverage trade-off than these controls.

  • 6.1 Strategic Recovery: 75.2% recovery rate is highest for DDO, compared with 70.1% for DivFreq and 69.7% for DPO.The probe replaces an interior action in successful BabyAI trajectories and measures success after rolling out from the edited prefix.
  • 6.2 Imitation Control: DDO retains a 24–32 pp task-success advantage over successful-only imitation under matched optimizer-step and 7.30× exposure conditions.Successful-only imitation broadens coverage, but DDO achieves higher task success in both comparisons.
  • 6.3 Decoding Control: At T=1.5, DPO reaches an ESD of 0.39, close to DDO’s 0.40, while DDO retains a 9 pp task-success advantage.Increasing DPO’s sampling temperature raises coverage but lowers task success.

7 Conclusion

DDO combines state-aligned branch collection with reference-relative training to preserve multiple successful strategies. Across the evaluated benchmarks, it delivers the strongest joint task success and successful-strategy coverage among the compared methods.

  • 7 Conclusion: DDO consists of Divergence-Tree Collection and the Reference-Relative Target-Odds Objective.DTC builds state-aligned branch sets with per-branch outcome labels, while RTO trains toward reference-relative targets over successful branches.
  • 7 Conclusion: Across all benchmarks, DDO achieves the strongest joint performance in task success and successful strategy coverage among compared post-training methods.It also achieves the highest recovery rate in the local action replacement evaluation.

Limitations

The experiments are limited to environments where states can be reconstructed and alternative branches can be executed from shared decision states under a common success predicate.

  • Limitations: The evaluation excludes continuous control, partial observability, stochastic dynamics, multimodal observations, and real-world agents with non-reversible side effects.Extending DTC beyond exact state reconstruction would require approximate state-aligned branch sets.

Ethical Considerations

The reported work uses licensed models and benchmark environments for academic, non-commercial evaluation, while also examining DDO across model-scale settings on BabyAI.

  • Ethical Considerations: The models and benchmark environments are used under their respective licenses for academic, non-commercial evaluation.Qwen models use Apache 2.0; BabyAI uses BSD 3-Clause; BabaIsAI and WebShop use MIT-based licensing.
  • Ethical Considerations: Reducing the expert model from 122B to 35B changes task success from 0.92 to 0.87, H-ESD from 0.45 to 0.41, and ESD from 0.52 to 0.48.The number of completed alternative branches remains similar: 3,329 versus 3,406.
  • Ethical Considerations: For the 1.7B target model, DDO improves task success, H-ESD, and ESD over its Reference by 0.10, 0.04, and 0.03, respectively.For the 4B target model, the corresponding gains are 0.06, 0.03, and 0.03.
  • Ethical Considerations: DDO stays above the other plotted methods across the sampled rollout-budget range on BabyAI, BabaIsAI, and WebShop.Figure 6 tracks the cumulative number of unique successful trajectories found within the first N rollouts.

A.3 Shared Cross-Task Post-Training

The shared cross-task setting tests whether DDO’s gains extend beyond task-specific adapters and whether its target-distribution and margin-sharpness choices affect performance. Across the reported benchmarks and ablations, DDO improves success and successful-strategy coverage, with α = 0.5 strongest in the target comparison and little sensitivity to β.

  • Target Distribution: α = 0 targets a uniform distribution over observed successful branches, whereas α = 1 preserves the reference distribution restricted to successful branches.The intermediate α = 0.5 contracts reference log-odds halfway toward uniformity.
  • Margin Sharpness: DDO varies little with margin sharpness across β values from 0.1 to 1.0: success changes mildly and H-ESD remains nearly unchanged after rounding.β controls the sharpness of the pairwise margin loss.
  • Main Benchmark Results: DDO leads benchmark averages on success rate, H-ESD, and ESD across BabyAI, BabaIsAI, and WebShop.The reported task-level results show joint success and coverage gains across navigation, rule manipulation, and shopping settings.

B Additional Method Details

The implementation represents executable actions through parsed model outputs, constructs benchmark-specific trajectory classes, and evaluates matched models and collection costs across the three environments. DTC uses bounded divergence and alternative-action budgets, while its collection cost is lower than the corresponding Without DTC setting on BabyAI and BabaIsAI.

  • Action Parsing: Each benchmark parser extracts an executable action from the LLM’s complete output, while DDO assigns probability to the full output sequence.Branch outcomes are determined by rollout execution from the parsed action.
  • Divergence-Tree Collection: DTC selects interior divergence points across trajectory depths and retains alternative outputs whose parsed actions differ from the source action.It uses at most 5 divergence points and 3 alternatives, with unsuccessful or unrecoverable branches handled by the collection procedure.
  • Trajectory Classes: BabyAI and BabaIsAI normalize trajectory classes by removing state-preserving action patterns, while WebShop combines purchase realization, selected options, and normalized trajectory structure.These rules define which successful trajectories count as distinct coverage strategies.
  • Models and Optimization: DDO, DPO, DivFreq, DivProb, TieDPO-RK, and TieDPO-Dav share the same SFT reference adapter, which also serves as the frozen reference model.The target models use Qwen3-1.7B and preference adapters under the reported optimization setup.
  • Collection Cost: DTC uses 44% of expert requests, 38% of expert tokens, and 75% of environment steps used by Without DTC per unique successful trajectory on BabyAI and BabaIsAI.RTO requires no additional expert calls or environment interactions.

C.4 Evaluation and Ablation Protocols

The evaluation fixes seeds, decoding, rollout budgets, and benchmark-specific coverage procedures, then compares DDO components, imitation, and decoding controls under matched protocols. The appendix examples show a shared thought-action format whose parsed action is executed in each environment.

  • Evaluation Protocol: BabyAI and BabaIsAI use 50 rollouts for success and 20 rollouts per seed for H-ESD and ESD, while WebShop reports success over 50 sessions.WebShop coverage uses sessions 500, 501, and 502 with 20 rollouts per session.
  • Component Comparison: The component comparison matches training budgets, contrasting state-aligned DTC branch sets with same-state comparisons derived from separately sampled rollouts.This isolates the contribution of the data-construction and optimization components.
  • Strategic Recovery: The strategic recovery probe replaces one interior action in successful BabyAI rollouts and measures success after continuing from the edited prefix.The balanced probe uses up to 30 successful rollouts per method-task pair.
  • Imitation Control: The imitation control continues SFT on all successful DTC branches from the same task-specific SFT initialization, with matched and full-epoch exposure conditions.Metrics use the two-benchmark aggregate.
  • Decoding Controls: The decoding ablation uses a three-benchmark macro average, with DPO temperatures from 0.0 to 1.5 and DDO sampled decoding at T = 0.6.BabyAI and BabaIsAI use epoch-5 policies, while WebShop uses epoch-15 policies.
  • Example LLM Calls: All three benchmarks use a thought-action response format in which the model reasons over textual observations before emitting one executable action.BabyAI exposes grid observations and admissible actions; BabaIsAI additionally exposes active rules; WebShop exposes page contents and interaction options.
Loading 2609.10052v1…