Source-linked AI summary
Cross-Relational Preference Learning for Better LLM Instruction Following
Runsheng Li, Kai Sun, Bin Shi, Bo Dong
TL;DR
LLMs remain limited on complex instructions, while existing preference-learning methods often overlook relationships among permissible response spaces. CRPL constructs relationship-aware preference data through targeted perturbation, cross-region sampling, and atomic verification, and reports consistent gains across methods, backbones, and four benchmarks.
Problem
Existing preference-learning approaches often fail to account for relationships between permissible response spaces of different instructions, limiting coverage of constraint variations.
Method
CRPL uses Cross-Relationship Perturbation, Cross-Region Pair Sampling, and atomic constraint-based verification to construct diverse, rigorously checked preference pairs.
Results
Across multiple preference-learning methods, LLM backbones, and four instruction-following benchmarks, CRPL consistently improves instruction-following performance and generalizes strongly.
Takeaways & Limitations
Explicitly modeling fine-grained response-space relationships provides preference data covering a wide spectrum of constraint variations.
Takeaways & Limitations
Further analysis was limited by computational resources to Qwen2.5-7B-Instruct with offline training and IFEval as the training set.
Abstract
from arXiv · showhide
Large Language Models (LLMs) still exhibit limited capability in following complex instructions. While existing approaches often rely on preference learning to enhance this ability, they typically overlook the relationships between the permissible response spaces of different instructions, which restricts a model to align with subtle and diverse constraint variations. To address this, we propose Cross-Relational Preference Learning (CRPL), a novel framework for constructing preference data that explicitly models inter-instruction relationships through two key techniques: Cross-Relationship Perturbation and Cross-Region Pair Sampling. This enables the generation of more diverse preference data that captures a wide spectrum of constraint variations. Additionally, we introduce an atomic constraint-based verification mechanism to rigorously assess response satisfaction, ensuring high-quality preference pair construction. Extensive experiments across multiple preference learning methods (e.g., DPO, KTO), LLM backbones and four instruction-following benchmarks demonstrate that our approach achieves substantial improvements over prior baselines and exhibits strong generalization.
1 Introduction
Complex instruction following remains difficult because existing preference-data methods often overlook relationships between permissible response spaces. CRPL addresses this by generating relationship-aware, diverse preference pairs and verifying each atomic constraint.
- Complex instructions remain challenging for LLMs because they involve multiple constraints requiring precise adherence.
- Existing approaches construct preference pairs through instruction synthesis, response correction, or constraint perturbation, but often overlook inter-instruction response-space relationships.
- CRPL models containment, partial overlap, and disjoint relationships between original and perturbed instruction response spaces.
- Cross-Relationship Perturbation steers constraint modifications toward a target relationship, enabling variants spanning all three relationship types.
- Cross-Region Pair Sampling covers disjoint and intersecting regions of combined response spaces so preference pairs represent the full relationship spectrum.
- Atomic constraint-based verification separately checks each constraint to ensure sampled positive responses satisfy the instruction.
- Across multiple preference-learning methods, LLM backbones, and four benchmarks, CRPL consistently improves instruction-following performance and generalization.
2 Related Work
Preference learning methods optimize LLM outputs toward preferred behavior, while instruction-following research increasingly focuses on constructing high-quality preference data. CRPL differs by explicitly modeling relationships between original and perturbed instruction response spaces.
- 2.1 Preference Learning: Preference learning directly optimizes model output distributions toward human preferences, with methods including DPO, KTO, and SimPO.
- 2.1 Preference Learning: Recent methods adapt DPO or iterative optimization to capture fine-grained distinctions between instructions and outputs.
- 2.2 Preference Data Construction for Instruction-Following: Instruction-following preference data is commonly built through rejection sampling, where synthesized instructions and verified responses form preference pairs.
- 2.2 Preference Data Construction for Instruction-Following: Other approaches correct negative responses or remove constraints to create comparable positive and negative responses.
- 2.2 Preference Data Construction for Instruction-Following: CRPL differs from prior approaches by explicitly modeling relationships between original and perturbed response spaces to increase preference-pair diversity.
3 Problem Definition
Complex instruction following is formulated as conditional text generation under a set of specific constraints. The objective is to generate a response satisfying every constraint, assessed through constraint-level verification.
- An instruction x_j entails a constraint set C_j containing distinct requirements such as length, keyword, or formatting constraints.
- The model must generate a response y_j that satisfies all constraints in C_j.
- The verification function V(y, c_i) returns 1 when response y satisfies constraint c_i and 0 otherwise.
4 Method
CRPL constructs preference data by perturbing instructions to target response-space relationships and sampling verified responses across the resulting regions. Its pipeline combines relationship-specific perturbation, cross-region pairing, atomic verification, and compatibility with multiple preference-optimization methods.
- Cross-Relationship Perturbation: CRPL decomposes instructions into constraints and generates perturbations for containment, partial-overlap, and disjoint response-space relationships.The perturbation process iterates over constraints and predefined relationships to produce distinct perturbed instruction sets.
- Cross-Region Pair Sampling: For each relationship, Cross-Region Pair Sampling derives preference pairs from sub-regions of the combined response space of original and perturbed instructions.For partial overlap, the regions include responses satisfying only the original instruction, only the perturbed instruction, or both.
- Cross-Region Pair Sampling: The sampling procedure pairs responses from contrasting regions so positives and negatives distinguish different patterns of constraint satisfaction.Responses from exclusive regions are paired with negatives from the opposite exclusive region, while responses satisfying both instructions also contribute to the pair construction.
- Atomic Constraint-based Verification: Atomic constraint-based verification uses dedicated verification functions to ensure sampled positives satisfy their designated constraints.Each function evaluates an individual constraint, and responses classified as positive are used in the preference data.
- Framework Compatibility: The resulting preference pairs are compatible with DPO, KTO, and an online variant of DPO.CRPL is presented as a general-purpose preference-pair generator rather than a method limited to one optimization objective.
5 Experiments
Experiments evaluate CRPL across datasets, preference-learning methods, backbones, perturbation relationships, sampling regions, and verification strategies. Results show broad performance gains, stronger coverage of perturbed constraints, and benefits from relationship diversity, cross-region sampling, and atomic verification.
- Main Results: CRPL improves instruction-following performance across both DPO and KTO training methods and all four test sets.With Qwen2.5-7B-Instruct under DPO, OurIFEval surpasses AutoIF by 6.9% in average accuracy across the four datasets.
- Main Results: CRPL maintains performance improvements across different LLM backbones and in online training, where UltraIF and SPAR underperform the pretrained backbone.The authors attribute the online-training gains to diverse sampling and atomic verification producing higher-quality preference pairs.
- Main Results: As training data grows, CRPL shows a clear upward performance trend, whereas other sampling methods exhibit limited gains.The comparison is conducted under both DPO and KTO, indicating that gains reflect data quality and informative preference signals in addition to quantity.
- Ablation Study: Removing any relationship type degrades performance, with disjoint relationships contributing the strongest contrastive signal.The ablation evaluates containment, partial overlap, and disjoint relationships on IFEval.
- Ablation Study: Removing any cross-region pair type reduces performance, showing that preference pairs must cover multiple regions to preserve sampling diversity.The partial-overlap ablation removes each of four pair types in turn.
- Ablation Study: Atomic Constraint-based Verification achieves the lowest misjudgment rate, zero false positives, and the highest downstream performance among compared verification methods.It is compared with LLM-as-Judge and Instruction-level Verification.
6 Conclusion
CRPL enhances instruction following by modeling fine-grained response-space relationships and constructing diverse, rigorously verified preference data. Experiments across training methods and backbones show strong benchmark performance.
- CRPL models fine-grained relationships between instructions’ response spaces to capture diverse constraint variations.
- Atomic constraint-based verification rigorously ensures the quality of sampled preference pairs.
- Experiments across diverse training methods and LLM backbones demonstrate strong performance on instruction-following benchmarks.
A.1 Datasets
The evaluation uses instruction-following benchmarks spanning verifiable constraints, progressively difficult multi-level constraints, and newer challenging constraint types. Their metrics assess accuracy at instruction and prompt levels where specified.
- IFEval contains approximately 540 instructions covering 25 types of verifiable constraints, including format, length, capitalization, and content.
- IFBench contains 58 constraint types and uses strict prompt- and instruction-level accuracy metrics.
- FollowBench evaluates content, situation, style, format, and example constraints through progressively increasing difficulty levels.
A.2 Baselines
The baselines include rejection-sampling, correction, code-verification, self-alignment, tree-search, and rule-plus-LLM validation approaches for constructing preference data or evaluating instruction following.
- Rejection sampling pairs verified correct and incorrect responses, with method-specific sampling choices for offline and online DPO comparisons.
- From Complex to Simple uses teacher correction, while AutoIF generates instructions and verification code for rejection-sampling preference data.
- UltraIF and Spar construct online-DPO preference data through iterative synthesis or tree-search self-correction, while VerIF combines code verification with LLM validation.
A.3 Training Configuration
Training uses LoRA adapters for efficient preference-learning experiments, while evaluation samples multiple responses per test instance and averages their scores.
- LoRA adapters are used for efficient training across preference-learning methods.
- Evaluation uses temperature 0.7, maximum generation length 4096, and four sampled responses per test instance.
- The reported test-set result is the average score across the four sampled responses.
B Additional Experimental Results
Additional materials specify prompts for decomposing instructions, generating atomic-constraint evaluators, and perturbing relationships between valid response spaces.
- Backbone Generalization: CRPL improves instruction-following performance on larger Qwen2.5-14B-Instruction and Qwen3-8B backbones under both DPO and KTO training.The reported gains extend beyond the primary backbone evaluation.
- Constraint Verification: The decomposition prompt extracts objectively verifiable hard constraints, including language, length, keyword, and formatting requirements.These constraints are intended for programmatic compliance checking.
- Constraint Verification: A separate prompt generates an evaluation function for one extracted constraint while ignoring all other prompt constraints.This isolates atomic-constraint verification during response assessment.
- Relationship Perturbation: The perturbation prompts cover containment in both directions, partial overlap, and disjoint relationships between valid response spaces.Together, these prompts operationalize multiple cross-instruction relationship types.