PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing LLM Reasoning Generalists with Preference Trees

Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, Maosong Sun

arXiv:2404.02078v1cs.AIcs.CLcs.LG

TL;DR

Open-source LLMs still lag proprietary models on broad reasoning, motivating a dataset and training approach for reasoning generalists. The paper introduces UltraInteract preference trees and Eurus models, then shows strong cross-task performance and a reward-modeling objective tailored to reasoning. Eurus achieves the best overall performance among comparable open-source models, while preference-learning results indicate that algorithm suitability differs between reasoning and general conversation.

  • Problem

    Open-source models fall substantially short of proprietary models in broad capabilities across diverse challenging reasoning tasks.

  • Method

    The paper builds UltraInteract preference trees containing diverse reasoning trajectories, multi-turn feedback interactions, and correct-incorrect pairs for fine-tuning, preference learning, and reward modeling.

  • Results

    Eurus achieves the best overall performance among similar-sized open-source models, with Eurus-70B outperforming GPT-3.5 Turbo and preference learning improving performance with KTO and NCA but not DPO.

  • Takeaways & Limitations

    UltraInteract enables strong open-source reasoning generalists and supports analysis showing that absolute reward values matter for reasoning preference learning.

Abstract

from arXiv · show

We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks covering mathematics, code generation, and logical reasoning problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a comprehensive benchmarking across 12 tests covering five tasks, and achieves a 33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging benchmarks, substantially outperforming existing open-source models by margins more than 13.3%. The strong performance of Eurus can be primarily attributed to UltraInteract, our newly-curated large-scale, high-quality alignment dataset specifically designed for complex reasoning tasks. UltraInteract can be used in both supervised fine-tuning and preference learning. For each instruction, it includes a preference tree consisting of (1) reasoning chains with diverse planning strategies in a unified format, (2) multi-turn interaction trajectories with the environment and the critique, and (3) pairwise data to facilitate preference learning. UltraInteract allows us to conduct an in-depth exploration of preference learning for reasoning tasks. Our investigation reveals that some well-established preference learning algorithms may be less suitable for reasoning tasks compared to their effectiveness in general conversations. Inspired by this, we derive a novel reward modeling objective which, together with UltraInteract, leads to a strong reward model.

1 Introduction

Open-source reasoning models still lag proprietary systems across diverse challenging tasks, motivating Eurus and UltraInteract. Eurus achieves strong cross-domain results, while UltraInteract supports both fine-tuning and preference learning.

  • Open-source models remain far behind proprietary models in broad capabilities across diverse challenging reasoning problems.
  • Eurus achieves state-of-the-art overall performance among open-source models across mostly out-of-distribution reasoning benchmarks.Eurus-70B is comparable to GPT-3.5 Turbo on TheoremQA and LeetCode Contest.
  • UltraInteract contains 86K instructions and 220K action pairs spanning mathematics, coding, and logical reasoning.Each instruction includes diverse planning strategies, multi-turn interaction trajectories, and paired correct and incorrect actions.
  • UltraInteract supports supervised fine-tuning and preference learning, with KTO and NCA improving performance while DPO hurts instruction-finetuned Eurus.
  • Reasoning performance correlates with chosen-data reward values, motivating a reward-modeling objective that increases chosen rewards and decreases rejected rewards.The resulting EURUS-RM-7B correlates better with human annotators than existing models on AutoJ and MT-Bench, including GPT-4.

2 ULTRAINTERACT: Tree-structured Alignment Data for Reasoning

UltraInteract represents reasoning supervision as preference trees spanning decomposition, tool use, environment interaction, critique, and correct-versus-incorrect actions. Ground-truth solutions support objective preference annotation and scalable preference learning.

  • Preference trees: Each UltraInteract instruction forms a preference tree whose root is the instruction and whose nodes are actions.Root-to-leaf trajectories support supervised fine-tuning on correct paths and preference learning from paired correct and incorrect nodes or trajectories.
  • Data construction: UltraInteract targets math problem-solving, code generation, and logical reasoning using challenging problems with ground-truth solutions.Ground-truth solutions provide oversight signals without relying on LLM-as-a-judge annotation.
  • Decomposition and interaction: At each turn, the actor decomposes problems, generates text or code actions under varied reasoning schemas, and uses an environment to execute them.
  • Decomposition and interaction: Multi-turn trajectories combine environment interaction with critique feedback so the actor can refine actions and correct previous errors.The illustrative trajectory shows step-by-step reasoning, execution, observations, and textual critique across turns.
  • Preference learning: Correct and incorrect actions are paired at each turn, with syntax-invalid instances excluded to reduce shortcut exploitation.
  • Data construction: Preference data collection uses repeated sampling and progressively different actor configurations when strong actors rarely produce correct actions.

3 EURUS: State-of-the-art Open LLMs in Reasoning

Eurus combines UltraInteract with supervised fine-tuning, preference learning, and reward modeling to build open-source reasoning models and a reward model. Its reward objective explicitly incorporates absolute reward values alongside relative preferences.

  • UltraInteract supports development of both the Eurus language-model suite and a reward model.
  • Eurus supervised fine-tuning uses correct UltraInteract actions, while preference learning uses multi-turn trajectory pairs alongside UltraFeedback.
  • EURUS-RM-7B is trained from multi-turn and single-turn UltraInteract pairs, augmented with UltraFeedback and UltraSafety.
  • The reward-model objective augments Bradley-Terry training with a term that raises chosen-action rewards and lowers rejected-action rewards for UltraInteract instances.Other datasets use the Bradley-Terry objective, and ablations demonstrate the importance of both terms.

4 Evaluation of EURUS-7B and EURUS-70B

EURUS is evaluated on single-turn and multi-turn reasoning benchmarks spanning coding, mathematics, logical reasoning, and instruction following. Both model sizes achieve strong overall performance among open-source models, while preference learning further improves especially mathematics and multi-turn ability.

  • Evaluation Setup: EURUS is evaluated with pass@1 accuracy across coding, mathematics, logical reasoning, and instruction-following benchmarks, including both single-turn and multi-turn settings.The evaluation covers HumanEval, MBPP, LeetCode, five math benchmarks, BBH-Hard, and IFEval.
  • Results: EURUS-7B and EURUS-70B achieve the best overall performance among open-source models of similar sizes.EURUS-7B also outperforms baselines five times larger, while EURUS-70B exceeds GPT-3.5 Turbo.
  • Results: EURUS has among the strongest instruction-following performance of general-purpose models and substantially exceeds specialized models.
  • Preference Learning: Preference learning with ULTRAINTERACT further improves performance, especially on mathematics and multi-turn evaluations.KTO and NCA consistently improve all five math benchmarks and multi-turn evaluations, whereas DPO hurts performance on most benchmarks.

5 Evaluation of EURUS-RM-7B

EURUS-RM-7B is evaluated as a reward model on out-of-distribution benchmarks and as a reranker for Mistral responses. It performs strongly against larger reward models, improves reasoning accuracy through reranking, and benefits from an objective tailored to hard reasoning problems.

  • Evaluation Setup: EURUS-RM-7B is evaluated on RewardBench, AutoJ, and MT-Bench, excluding RewardBench’s contaminated prior-sets split.The model is also tested for reranking Mistral-7B-Instruct-v0.2 responses.
  • Reward Modeling Results: EURUS-RM-7B achieves the strongest overall 7B reward-model performance, matching or exceeding much larger baselines and outperforming GPT-4 on certain tasks.
  • Reward Modeling Results: EURUS-RM-7B outperforms nearly all existing models across AutoJ splits, except GPT-4 on Coding.
  • Reward Modeling Results: Optimizing LDR improves reward-model reasoning ability, while BT modeling remains beneficial for general chatting and its reasoning effect varies.
  • Data Mixture: ULTRAINTERACT improves reward-model reasoning without sacrificing other abilities and can be mixed with UltraFeedback and UltraSafety to balance capabilities.
  • Reranking Results: EURUS-RM-7B consistently improves pass@1 accuracy across tasks through reranking and outperforms the 5× larger Starling-RM-34B.Performance scales well with the number of responses, except for a slight HumanEval decrease from 8 to 16 responses; Starling-RM-34B drops severely on HumanEval and hurts MATH accuracy.

6 Analysis

The analysis links preference-learning behavior to absolute rewards assigned to chosen solutions and motivates a reward objective beyond relative preference margins. Ablations show that ULTRAINTERACT’s structured designs help, but mixing it with other alignment data supports broader supervised performance.

  • Preference Learning Analysis: DPO decreases rewards for both chosen and rejected data, whereas KTO and NCA increase chosen rewards while decreasing rejected rewards.
  • Preference Learning Analysis: DPO optimizes relative chosen–rejected differences but overlooks absolute reward values, which may be less suitable when correct reasoning answers are comparatively scarce.
  • Preference Learning Analysis: Chosen-data rewards rank KTO > NCA > DPO at the final training step and positively correlate with performance trends.
  • Ablation Study: ULTRAINTERACT-only training lowers performance except on BBH, especially instruction following, supporting a mixture with other alignment data for all-around SFT.
  • Ablation Study: EURUS exceeds the ground-truth model on all tasks, supporting the value of ULTRAINTERACT’s divide-and-conquer and code-as-action designs.

7 Related Work

Prior work has advanced open-source specialization in mathematics and coding, but general reasoning remains challenging. Preference learning is established for alignment yet remains comparatively underexplored for complex reasoning, where DPO degradation has been reported.

  • Open LLMs in Reasoning: Open-source LLMs have progressed as mathematics and coding specialists, while mastering general reasoning remains challenging.
  • Preference Learning for Reasoning: Preference learning is prevalent for aligning language models, but its application to complex reasoning remains largely underexplored.
  • Preference Learning for Reasoning: Recent research has reported performance degradation when applying DPO to reasoning tasks.

8 Conclusion

The work advances open-source reasoning generalists through the ULTRAINTERACT dataset, EURUS models, and preference-learning analysis. It combines broad task coverage, curated oversight, decontamination, and large-scale action-pair data.

  • Contributions: ULTRAINTERACT and EURUS address math, coding, and logical reasoning through high-quality alignment data and reasoning-focused models.The dataset uses instructions from established datasets and includes preference-tree structures for reasoning supervision.
  • Task coverage: The dataset spans competition-level Python coding, mathematical reasoning, and multi-hop or logical reasoning tasks.Coding uses CodeContest and TACO; logical reasoning uses HotpotQA, StrategyQA, and ReClor, with HotpotQA converted to generation requiring Wikipedia search.
  • Action generation: Correct actions are generated from ground-truth annotations, with task-specific access designed to preserve complete reasoning chains.Coding exposes solution code, while tool-free math masks answer numbers before generation.
  • Evaluation hygiene: Decontamination removes instructions overlapping evaluation samples through exact substring matching for LeetCode and 8-gram matching elsewhere.The procedure compares ULTRAINTERACT instructions against relevant test sets and removes overlaps.
  • Dataset scale: ULTRAINTERACT contains 86K instructions and 220K action pairs, although pair counts do not equal turns or correct answers.The mismatch reflects limited ground-truth annotations and the unnecessary sampling of incorrect actions for some simple instructions.

B Additional Details on Training EURUS Models

The training setup applies separate supervised fine-tuning, preference-learning, reward-modeling, and pair-augmentation procedures. Pair augmentation expands cross-turn comparisons while limiting selected trajectories to control overfitting.

  • Supervised Fine-Tuning: Supervised fine-tuning runs for one epoch with a 2e-5 learning rate, 0.1 warmup ratio, and cosine scheduling.EURUS-7B and EURUS-70B use different mixtures of UltraChat, ShareGPT, and OpenOrca.
  • Preference Learning: Preference learning uses β=0.1, KTO λ+/λ−=1.33, one epoch, a 5e-7 learning rate, and cosine scheduling.The warmup ratio is 0.1.
  • Reward Modeling: Reward models train for one epoch with a 1e-5 learning rate, 0.1 warmup ratio, and cosine scheduling.
  • Pair Augmentation: Pair augmentation creates NxN cross-turn action pairs and retains only instructions with NxN ≤10.Same-turn pairs are removed because they already occur in multi-turn trajectories.

C Additional Evaluation Results of EURUS

Additional evaluations show that EURUS transfers to knowledge and conversation benchmarks while retaining strong reasoning-oriented performance. Its results are competitive with strong open-source systems and, in one setting, GPT-3.5 Turbo.

  • Evaluation setup: The evaluation reports MMLU and MT-Bench results, using 5-shot MMLU for STEM knowledge and MT-Bench for conversation abilities.Most evaluations use 0-shot chain-of-thought, with BBH using 3 shots and IFEval omitting chain-of-thought.
  • MMLU: EURUS outperforms coding- and math-specialized baselines on MMLU and exceeds the aligned versions of its Mistral and CodeLLaMA base models.EURUS-7B is comparable with top-performing OpenChat and Starling-LM, while EURUS-70B trails other general-purpose models.
  • MT-Bench: EURUS matches mainstream open-source general-purpose models on MT-Bench, and EURUS-70B-KTO reaches GPT-3.5 Turbo’s score.

D.1 Additional Results on Reranking

The reranking experiments evaluate reward models across coding and math test sets and support the proposed reward-modeling design. The reported findings emphasize broad accuracy and the effects of objective components and safety data.

  • Reranking results: EURUS reward models achieve the highest reranking accuracy across test sets and candidate counts, except at N=2 on HumanEval.
  • Objective ablations: LBT and LDR generally improve reranking, while removing either can prevent accuracy drops on HumanEval when N increases from 8 to 16.
  • Safety-data ablation: Removing UltraSafety yields higher reward-model accuracies than EURUS-RM-7B except on MBPP.
  • Reported evaluation: The full reranking results are reported in a table covering coding and math datasets.
Loading 2404.02078v1…