PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Relational Preference Learning for Better LLM Instruction Following

Runsheng Li, Kai Sun, Bin Shi, Bo Dong

arXiv:2608.29352v1cs.AI

TL;DR

LLMs remain limited in following complex instructions, while preference-learning methods often overlook relationships between instructions’ permissible response spaces. CRPL models those relationships when constructing preference data, and experiments report consistent instruction-following gains across four benchmarks, multiple preference-learning methods, and LLM backbones, with strong generalization.

  • Problem

    LLMs remain limited on complex multi-constraint instructions, while preference-learning methods often overlook relationships between different instructions’ permissible response spaces.

    Source: §1 · p. 1

  • Method

    CRPL constructs preference data by modeling response-space relationships between original and perturbed instructions, with atomic constraint-based verification of sampled responses.

    Source: §1 · p. 2

  • Results

    CRPL consistently improves instruction-following performance across multiple preference-learning methods, diverse LLM backbones, and four benchmarks, and exhibits strong generalization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results support CRPL as a generalizable preference-learning approach for instruction following across the tested methods and backbones.

    Source: §1 · p. 2, §6 · p. 7

    Further analysis is limited to Qwen2.5-7B-Instruct using offline training strategies with IFEval as the training set.

    Source: §5.3 · p. 6

Abstract

from arXiv · show

Large Language Models (LLMs) still exhibit limited capability in following complex instructions. While existing approaches often rely on preference learning to enhance this ability, they typically overlook the relationships between the permissible response spaces of different instructions, which restricts a model to align with subtle and diverse constraint variations. To address this, we propose Cross-Relational Preference Learning (CRPL), a novel framework for constructing preference data that explicitly models inter-instruction relationships through two key techniques: Cross-Relationship Perturbation and Cross-Region Pair Sampling. This enables the generation of more diverse preference data that captures a wide spectrum of constraint variations. Additionally, we introduce an atomic constraint-based verification mechanism to rigorously assess response satisfaction, ensuring high-quality preference pair construction. Extensive experiments across multiple preference learning methods (e.g., DPO, KTO), LLM backbones and four instruction-following benchmarks demonstrate that our approach achieves substantial improvements over prior baselines and exhibits strong generalization.

1 Introduction

CRPL constructs diverse preference data by explicitly modeling relationships between instructions’ permissible response spaces and verifying responses against atomic constraints. Experiments across preference-learning methods, LLM backbones, and four benchmarks show improved instruction following and strong generalization.

  • LLMs handle simple instructions well but remain limited on complex instructions involving multiple constraints [Pyatkin et al., 2025].Source: §1 · p. 1
  • Prior work used instruction-level rejection sampling [An et al., 2025; Dong et al., 2025] or constraint-level correction and random perturbations [Cheng et al., 2025; Jiang et al., 2025; Zhang et al., 2025; Huang et al., 2025].Source: §1 · p. 1
  • CRPL addresses preference-data diversity limits by explicitly modeling containment, partial-overlap, and disjoint relationships between instructions’ permissible response spaces.Source: §1 · pp. 1–2
  • Cross-Relationship Perturbation steers constraint modifications toward a target relationship, while Cross-Region Pair Sampling covers pertinent regions of the combined response space.For partial overlap, sampled positives span both instructions’ disjoint regions and their intersection.Source: §1 · p. 2
  • Atomic constraint-based verification applies a separate function to each constraint, unlike whole-pair or single-function verification used by UltraIF [An et al., 2025] and IOPO [Zhang et al., 2025].Source: §1 · p. 2
  • CRPL consistently improves instruction-following performance and generalizes across multiple preference-learning methods, diverse LLM backbones, and four benchmarks.Source: §1 · p. 2

2 Related Work

Related work advances preference optimization for instruction following and develops varied ways to construct preference data; this work distinguishes itself by explicitly modeling relationships between the response spaces of original and perturbed instructions.

2.1 Preference Learning

2.2 Preference Data Construction for Instruction-Following

  • Preference-data construction includes rejection sampling, ULTRAIF’s iterative synthesis and response evaluation [An et al., 2025], and AutoIF’s LLM-generated instructions with code-based response verification [Dong et al., 2025].High-quality preference data is described as critical for preference learning.Source: §2.2 · p. 2
  • Correction-based approaches form preference pairs by revising negative responses, using teacher correction in FCS [He et al., 2024], tree-search self-correction in SPAR [Cheng et al., 2025], or minimal edits in BMC [Jiang et al., 2025].Other methods perturb instructions: IOPO [Zhang et al., 2025] alters constraints, while MUSC [Huang et al., 2025] removes atomic constraints to create negative instructions.Source: §2.2 · p. 2
  • Unlike prior approaches, this work explicitly models relationships between original and perturbed instructions’ response spaces to diversify preference pairs across constraint variations and improve instruction following.Source: §2.2 · p. 3

3 Problem Definition

Complex instruction following is formulated as generating a response that satisfies every constraint entailed by an input instruction. A verification function evaluates whether a response satisfies an individual constraint.

3 Problem Definition

  • The model, parameterized by θ, aims to generate a response satisfying all constraints entailed by the input instruction.Constraints can specify requirements such as length limits, keyword inclusion, or formatting style.Source: §3 · p. 3
  • The verification function V(y, ci) returns 1 when response y satisfies constraint ci and 0 otherwise.Source: §3 · p. 3

4 Method

CRPL constructs preference pairs by perturbing instructions to create specified response-space relationships, sampling across the resulting regions, and verifying responses against atomic constraints.

4 Method

  • CRPL generates preference pairs by perturbing instructions across response-space relationships, then sampling across regions of the combined response spaces with atomic-constraint verification.The framework overview describes Cross-Relationship Perturbation followed by Cross-Region Pair Sampling.Source: §4 · p. 3, Figure 2 · p. 4

4.1 Cross-Relationship Perturbation

  • For each instruction, CRPL decomposes hard, executable constraints and generates variants whose response spaces have containment, partial-overlap, or disjoint relationships to the original.Each constraint is perturbed to produce variants for all three predefined relationships.Source: §4.1 · p. 3

4.2 Cross-Region Pair Sampling

  • Cross-Region Pair Sampling partitions combined response spaces into satisfaction regions and pairs exclusive-region positives with negatives from the opposite exclusive region for contrastive signals [Yan et al., 2024; Zhang et al., 2025].For partial overlap, the regions are responses satisfying only the original instruction, only the perturbed instruction, or both.Source: §4.2 · p. 3

Atomic Constraint-based Verification

  • Each atomic constraint gets an LLM-generated verification function that must execute and exceed 80% accuracy on six LLM-generated test cases or be regenerated; responses are positive when the verification result is 1.The mechanism evaluates response satisfaction at the individual-constraint level.Source: Atomic Constraint-based Verification · p. 4

4.3 Training Objectives

5 Experiments

CRPL improves instruction-following performance across offline and online training settings, with results indicating gains across test sets, backbones, and perturbed instruction variants. The experiments also show that diverse preference sampling and atomic verification contribute to preference-pair quality.

5.1 Setup

  • Experiments use IFEval, IFBench, 200 Cross-Relationship-Perturbed instructions, FollowBench, and Qwen2.5-7B and Llama-3.1-8B backbones.IFEval uses 433 training and 108 held-out test instructions; IFBench uses 2,000 sampled training instructions and its 294-instruction official test set. Comparisons include offline and online preference-optimization baselines.Source: §5.1 · pp. 4, 6

5.2 Main Results

  • Under Qwen2.5-7B DPO, OurIFEval surpasses AutoIF by 6.9% in average accuracy across four datasets.The authors also report gains across both DPO and KTO, all four test sets, and both evaluated backbones; online CRPL improves performance despite limited prior-baseline results without advanced-LLM supervision.Source: Table 1 · p. 5, §5.2 · p. 6

Performance on Different Relationship Types

  • CRPL improves performance on original instructions and all four perturbed types, whereas baselines show limited gains on perturbed variants.The comparison covers Rejection Sampling, From Complex to Simple, and AutoIF; Rejection Sampling improves more consistently across perturbations, while From Complex to Simple improves only on Containment-1.Source: Table 2 · p. 5, Performance on Different Relationship… · pp. 6–7

5.3 Further Analysis

  • With offline training on Qwen2.5-7B and IFEval, further analyses are limited by computational resources.This is the stated scope for the section’s subsequent analysis.Source: §5.3 · p. 6

Performance on Different Data Sizes

  • As training data grows, CRPL shows an upward performance trend under DPO and KTO, while other methods show limited gains.The authors interpret this as evidence that improvements reflect higher-quality data and more informative preference signals, not just greater data quantity.Source: Performance on Different Data Sizes · p. 6, Figure 3 · p. 7

Ablation Study

  • ACV yields the lowest misjudgment rate and zero false positives when evaluating responses against IFEval test instructions.The comparison is against LLM-as-Judge and Instruction-level Verification under the official IFEval validation protocol.Source: Table 5 · p. 7, Ablation Study · p. 7
  • Removing any relationship type degrades performance, with Disjoint contributing the most; excluding any cross-region pair type also reduces performance.For verification, ACV has the lowest misjudgment rate and zero false positives, and produces the highest-performing models among ACV, ILV, and LAJ.Source: Table 3 · p. 6, Ablation Study · p. 7
  • Training preference data constructed with ACV achieves higher performance than data constructed with ILV or LAJ.This further supports the authors’ claim that ACV better ensures preference-pair quality.Source: Ablation Study · p. 7

6 Conclusion

CRPL enhances LLM instruction following by modeling fine-grained response-space relationships across instructions to generate preference data covering diverse constraint variations. Atomic constraint-based verification ensures sampled-pair quality, and experiments report strong benchmark performance across training methods and LLM backbones.

6 Conclusion

  • CRPL models fine-grained response-space relationships across instructions to generate preference data spanning diverse constraint variations.Source: §6 · p. 7
  • Atomic constraint-based verification rigorously ensures the quality of sampled preference pairs.Source: §6 · p. 7
  • Experiments across diverse training methods and LLM backbones report strong performance on instruction-following benchmarks.Source: §6 · p. 7

A Experimental Setup · A.1 Datasets

The dataset suite uses IFEval and IFBench with strict prompt- and instruction-level accuracy, and FollowBench to assess generalization where code-based verification is not applicable.

A.1 Datasets

  • IFEval tests verifiable instruction following across approximately 540 instructions and 25 constraint types, using strict prompt- and instruction-level accuracy.The paper reports these metrics as Accins and Acccon.Source: §A.1 · p. 10
  • IFBench [Pyatkin et al., 2025] evaluates precise instruction following across 58 constraint types with verification functions, reporting strict prompt- and instruction-level accuracy.The paper reports these metrics as Accins and Acccon.Source: §A.1 · p. 10
  • FollowBench [Jiang et al., 2024] assesses generalization on instructions without applicable code-based verification, using GPT-4o-mini as evaluator.It covers five constraint types and progressively increases difficulty through a multi-level mechanism.Source: §A.1 · p. 10

A.2 Baselines

The baselines cover rejection sampling, teacher-corrected and verifier-based preference construction, and self-alignment methods; the paper uses official releases, code, datasets, or checkpoints for reproductions and comparisons where specified.

A.2 Baselines

  • Rejection sampling pairs verified correct and incorrect responses; offline DPO and KTO use DeepSeek v3.1 positives, while online DPO samples both from the previous iteration’s model.Negative responses for offline DPO and KTO come from the model being trained.Source: §A.2 · p. 10
  • From Complex to Simple [He et al., 2024] forms preference pairs through teacher correction of student outputs and was reproduced using its official preference data.Source: §A.2 · p. 10
  • AutoIF generates instructions and verification code with LLMs, then uses code-based rejection sampling to construct preference data; it was reproduced with official open-source code.Source: §A.2 · p. 10
  • UltraIF [An et al., 2025] iteratively synthesizes complex instructions and samples and evaluates responses to build online-DPO preference pairs; it was reproduced with its official dataset and code.Source: §A.2 · p. 10
  • Spar [Cheng et al., 2025] builds online-DPO preferences through tree-search self-correction; it was reproduced on Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct using official code.Source: §A.2 · p. 10
  • VerIF [Peng et al., 2025] combines rule-based code verification with LLM validation for RLVR; released model checkpoints were used for evaluation and comparison.Source: §A.2 · p. 10

A.3 Training Configuration · A.4 Generation Configuration

Training uses LoRA adapters for each preference learning method, while evaluation averages four sampled responses per test instance under fixed generation settings.

A.3 Training Configuration

  • Each preference learning method uses LoRA adapters, and online DPO shares the offline DPO settings.Training hyperparameters are listed in Table 6.Source: §A.3 · p. 10

A.4 Generation Configuration

  • Evaluation uses temperature 0.7 and a 4096-token maximum; each test instance contributes the average score of four sampled responses.Settings apply to each test set and backbone LLM.Source: §A.4 · p. 10

B Additional Experimental Results · C Prompts of CRPL

CRPL improves test-set results with DPO and KTO on larger Qwen backbones, and the appendix provides prompts for decomposition, atomic-constraint evaluation, and four relationship perturbations.

B Additional Experimental Results

  • CRPL achieves significant test-set improvements with DPO and KTO on Qwen2.5-14B-Instruction and Qwen3-8B.The passage refers to results in Table 7 but provides no values.Source: B Additional Experimental Results · p. 10

C Prompts of CRPL

  • Figure 4 presents the prompt for instruction decomposition.Source: Figure 4 · p. 11
  • Figure 5 presents the prompt for generating evaluation functions for atomic constraints.Source: Figure 5 · p. 11
  • Figure 6 presents a prompt for perturbation where the original instruction’s valid response space contains the perturbed instruction’s.Source: Figure 6 · p. 12
  • Figure 7 presents a prompt for perturbation where the perturbed instruction’s valid response space contains the original instruction’s.Source: Figure 7 · p. 12
  • Figure 8 presents the prompt for partial-overlap relationship perturbation.Source: Figure 8 · p. 13
  • Figure 9 presents the prompt for disjoint relationship perturbation.Source: Figure 9 · p. 13

Related papers

Loading 2608.29352v1…