PaperlayerView exact revision on arXiv

Source-linked AI summary

MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models

Junjie Ye, Caishuang Huang, Zhuohan Chen, Wenjie Fu, Chenyuan Yang, Leyi Yang, Yilong Wu, Peng Wang, Meng Zhou, Xiaolong Yang, Tao Gui, Qi Zhang, Zhongchao Shi, Jianping Fan, Xuanjing Huang

arXiv:2505.07591v2cs.CLcs.AI

TL;DR

Instruction-following research has limited evaluation dimensions and little guidance about the mechanisms behind improvement. MulDimIF introduces a multi-dimensional constraint framework and controllable code-verifiable data pipeline, finding strong difficulty-related performance differences and improved instruction following after training without compromising general performance.

  • Problem

    Existing research focuses mainly on constraint categories, providing limited evaluation dimensions and little insight into what drives instruction-following performance.

    Source: §1 · pp. 1–2

  • Method

    MulDimIF combines three constraint patterns, four categories, four difficulty levels, and a pipeline of constraint expansion, conflict detection, and instruction rewriting to create 9,106 code-verifiable instances.

    Source: §1 · p. 2

  • Results

    Average accuracy decreases from 80.82% at Level I to 36.76% at Level IV, while training on framework-generated data improves instruction following without sacrificing general performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Attention-module updates primarily account for the gains by strengthening models’ recognition of and adherence to specified constraints.

    Source: §1 · p. 2

    Evaluation before and after reinforcement learning uses only a limited set of representative datasets, and domain-specific datasets are not explored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Instruction following refers to the ability of large language models (LLMs) to generate outputs that satisfy all specified constraints. Existing research has primarily focused on constraint categories, offering limited evaluation dimensions and little guidance for improving instruction-following abilities. To address this gap, we introduce MulDimIF, a multi-dimensional constraint framework encompassing three constraint patterns, four constraint categories, and four difficulty levels. Based on this framework, we design a controllable instruction generation pipeline. Through constraint expansion, conflict detection, and instruction rewriting, we construct 9,106 code-verifiable samples. We evaluate 18 LLMs from six model families and find marked performance differences across constraint settings. For instance, average accuracy decreases from 80.82% at Level I to 36.76% at Level IV. Moreover, training with data generated by our framework significantly improves instruction following without compromising general performance. In-depth analysis indicates that these gains stem largely from parameter updates in attention modules, which strengthen constraint recognition and adherence. Code and data are available in https://github.com/Junjie-Ye/MulDimIF.

1 Introduction

MulDimIF addresses limited evaluation dimensions and weak mechanistic insight in instruction-following research with a multi-dimensional constraint framework and controllable data-generation pipeline. Evaluations reveal steep difficulty-related performance declines, while targeted training improves instruction following without compromising general performance.

Motivation

  • Existing benchmarks emphasize constraint-category diversity but use narrow evaluation dimensions, while training studies rarely examine the mechanisms behind gains.These limitations motivate MulDimIF’s combined evaluation and improvement framework.Source: §1 · pp. 1–2

Framework

  • MulDimIF organizes instructions by three constraint patterns, four categories, thirteen subcategories, and four difficulty levels.The patterns are example, listing, and incorporation; categories are content, language, format, and length.Source: Figure 1 · p. 1, §1 · p. 2

Data generation

  • The pipeline expands constraints, detects redundancy or conflict, and rewrites instructions into diverse constraint-rich variants, yielding 9,106 code-verifiable instances.These stages support controlled transformation of initial instructions.Source: §1 · p. 2

Evaluation findings

  • 80.82% average accuracy at Level I falls to 36.76% at Level IV, with the best model scoring 67.50% overall.The evaluation covers 18 LLMs across six model families and finds strong variation across constraint forms.Source: §1 · p. 2

Training

  • Training on framework-generated data significantly improves instruction following without sacrificing general performance.The authors train models with GRPO on the generated data.Source: §1 · p. 2

Mechanistic analysis

  • Parameter-level analysis indicates that gains primarily arise from attention-module updates that better align model focus with specified constraints.The analysis links improvement to stronger constraint recognition and adherence.Source: §1 · p. 2

2 Related Works

Prior work evaluates instruction following with diverse benchmarks and improves it through optimization and curriculum methods, but often uses constrained prompts or lacks mechanistic analysis. MulDimIF responds with multi-dimensional constraint modeling, controllable data construction, and attention-focused analysis.

Evaluation of Instruction Following

  • Many instruction-following benchmarks rely on templated or highly constrained prompts, limiting real-world diversity and fine-grained evaluation.Examples include IFEval, FollowEval, FollowBench, Multi-IF, InfoBench, and CIF-Bench.Source: §2 · p. 2

MulDimIF’s approach

  • MulDimIF expands evaluation and data construction through three constraint patterns, four categories, four difficulty levels, and a pipeline for expansion, conflict detection, and rewriting.The pipeline is designed to increase instruction diversity and complexity while enabling code verification.Source: §2 · p. 2, Figure 2 · p. 3

Improving of Instruction Following

  • Prior improvement methods include preference optimization, cross-dataset preference aggregation, and curriculum learning for progressively harder constraint handling.The related methods include PPO, DPO, IOPO, and Coninfer.Source: §2 · p. 2
  • Existing training approaches often lack in-depth analysis of the model characteristics driving improvements, limiting interpretability and generalizability.MulDimIF’s parameter-level analysis points to attention mechanisms as a major source of improvement.Source: §2 · p. 3

3 Approaches

MulDimIF organizes instruction-following constraints by presentation pattern, constraint category, and difficulty, then generates code-verifiable variants through a controllable pipeline. The resulting data support systematic evaluation across dimensions.

3.1.1 Constraint Pattern

  • Three constraint patterns—example, listing, and incorporation—capture distinct ways of presenting requirements in instructions.Example uses contextual question-answer pairs, listing states requirements point by point, and incorporation embeds constraints directly in the instruction.Source: §3.1.1 · pp. 3–4

3.1.2 Constraint Category

  • Four categories—content, format, language, and length—comprise thirteen subcategories for fine-grained constraint analysis.Examples include keywords or punctuation, XML or JSON structure, specified output languages, and paragraph-, sentence-, or word-level limits.Source: §3.1.2 · p. 4

3.1.3 Constraint Difficulty

  • Four difficulty levels increase from one constraint type with one or two elements at Level I to four types with four to eight elements at Level IV.Levels II and III contain two to four and three to six individual constraint elements, respectively.Source: §3.1.3 · p. 4

3.2 Instruction Generation Pipeline

  • The controllable pipeline expands constraints, detects missing or conflicting requirements, and rewrites instructions into selected patterns.Expansion adds constraints from uncovered categories; invalid or conflicting instructions are discarded, while rewriting can add three matching question-answer examples.Source: §3.2 · p. 4
  • The pipeline produces 9,106 data instances, each manually reviewed and paired with Python validation code for automated evaluation.The data are sampled from ShareGPT and organized into training and test distributions summarized in Table 1.Source: Table 1 · p. 4, §3.2 · p. 5

4 Evaluations of LLMs

The study evaluates 18 LLMs from six model families under a multi-dimensional instruction-following test. Performance varies substantially by constraint pattern, category, difficulty, scale, and reasoning mode.

4.1 Models

  • The evaluation covers 18 LLMs from six families, including open- and closed-source models and Qwen3 variants in both direct and reasoning modes.Models use their built-in chat templates or official APIs, with greedy decoding and only final answers evaluated for reasoning models.Source: §4.1 · p. 5, §4.2 · p. 5

4.3 Main Results

  • Models generally perform best on example constraints and worst on incorporation constraints, while content constraints are easier than language- and length-related constraints.The pattern is consistent with the usefulness of contextual examples and the difficulty of following free-form requirements.Source: §4.3 · p. 5
  • Average accuracy falls from 80.82% at Level I to 36.76% at Level IV, indicating considerable challenges with multiple or more complex constraints.Source: §4.3 · p. 6
  • Larger models usually follow instructions more reliably, especially at Level IV, but Qwen3 and GPT include exceptions such as Qwen3-32B-Direct. underperforming Qwen3-14B-Direct. and GPT-4o underperforming GPT-4o-Mini.The authors associate these deviations with a possible alignment tax.Source: §4.3 · p. 6
  • Reasoning mode mainly raises weaker dimensions such as incorporation, length, and Level IV, with little benefit for stronger dimensions such as example, content, and Level I.DeepSeek-R1-Distill-LLaMA-Instruct-8B also performs below LLaMA3.1-Instruct-8B.Source: §4.3 · p. 6

5 Improvements of LLMs

GRPO training on MulDimIF-generated data improves instruction following while preserving general capabilities. Parameter analysis and token-level case studies link the gains to stronger attention toward constraint-relevant information.

5 Improvements of LLMs

  • The training set contains 7,906 single-turn constraint-rich instructions, and GRPO is applied to six LLMs with fewer than 14 billion parameters.Training uses a constraint-satisfaction reward, one epoch, and greedy decoding for evaluation.Source: §5 · p. 6, §5.2 · p. 6

5.3 Results and Analysis

  • GRPO substantially improves performance on the custom test set and transfers to out-of-domain and multi-turn instruction-following benchmarks.LLaMA3.1-Instruct-8B notably outperforms other models after training, despite training using only single-turn data.Source: §5.3 · p. 6, Figure 3 · p. 7

5.1 Test Set

  • Post-GRPO models maintain parity on general benchmarks and sometimes improve, indicating no degradation of general-purpose performance.Evaluation covers instruction following, knowledge, reasoning, and coding using the custom test set, IFEval, Multi-IF, MMLU, GSM8K, MATH, HumanEval, and MBPP.Source: §5.1 · p. 6, §5.3 · p. 7

5.3 Results and Analysis

  • The largest parameter updates occur in attention modules and are distributed relatively uniformly across layers, suggesting global refinement of attention mechanisms.The analysis links these updates to better identification and focus on critical input information.Source: §5.3 · p. 7, Figure 4 · p. 7
  • After GRPO, constraint-related tokens become more important, irrelevant tokens less influential, and core problem components remain largely unchanged.This pattern is presented as evidence that models prioritize constraints without sacrificing understanding of the overall input.Source: §5.3 · p. 8, Table 4 · p. 8

6 Conclusion

MulDimIF combines multi-dimensional constraint categorization with controllable data generation, evaluation across 18 LLMs, and GRPO-based improvement. The resulting models improve instruction following while preserving general capabilities, with gains associated with greater sensitivity to constraint-relevant information.

6 Conclusion

  • MulDimIF categorizes instructions by constraint pattern, category, and difficulty, then generates variants through expansion, conflict detection, and rewriting.The pipeline produces 9,106 instances for evaluation and training.Source: §6 · pp. 8–9
  • Models trained with MulDimIF data achieve significant instruction-following improvements while preserving general capabilities.Parameter-level analysis and case studies suggest that gains primarily reflect increased sensitivity to constraint-relevant information.Source: §6 · p. 9

Limitations

The authors identify two limitations: evaluation before and after RL uses only a limited set of representative datasets, and domain-specific applications are not explored.

Limitations

  • Limited representative datasets may not fully capture changes in models’ original abilities after RL.The reported results nevertheless show largely stable performance on widely used benchmarks.Source: Limitations · p. 9
  • The method’s effects on domain-specific datasets, including reasoning and coding, remain unexplored.The authors note that their constraint-based conversion could potentially be applied to other domains, but case studies do not establish those gains.Source: Limitations · p. 9

A Comparison of MulDimIF with Existing Approaches

MulDimIF combines broad, automated, code-verifiable data construction with manual quality control and multi-dimensional constraint coverage. Its generated examples support more fine-grained analysis than templated instructions.

A Comparison of MulDimIF with Existing Approaches

B Details about Manual Review

  • Six annotators manually verified and refined all 9,106 instances and validation code over four weeks, with cross-validation by at least two annotators.Source: B Details about Manual Review · p. 12
  • Validation code was generated by DeepSeek-V3, reviewed for correctness and coverage, and independently approved or revised by a second annotator.Source: B Details about Manual Review · p. 12

C Examples for Different Forms of Constraints

D Detailed Information of Models

The evaluated model set spans seven families and includes open- and closed-source systems, with Qwen3 assessed in both non-reasoning and reasoning modes. Training results show gradual convergence and steadily improving MulDimIF performance.

D Detailed Information of Models

E.1 Details of Training Process

Training shows gradual convergence alongside steadily improving performance on MulDimIF.

E.1 Details of Training Process

  • Training yields gradual convergence and a consistently upward trend in MulDimIF performance.Source: §E.1 · p. 13

E.2 Details of Final Results

The final-results materials report broad gains from GRPO using the generated data, while the accompanying prompts operationalize constraint expansion, conflict detection, and pattern-specific rewriting.

E.2 Details of Final Results

  • GRPO substantially enhances model capabilities across all reported test sets, according to the detailed before-and-after results in Tables 9–14.The tables report performance differences between post-GRPO and pre-GRPO models, with positive and negative changes highlighted.Source: §E.2 · p. 13

F Prompts for Instruction Generation

  • The instruction-generation materials include examples spanning content, language, length, format, and keyword constraints across varied questions.Examples require capitalization, word limits, keywords, Markdown headings, and other response-format properties.Source: F Prompts for Instruction Generation · pp. 16–17, Table 6 · p. 18

Prompt for Constraint Expand

  • Constraint expansion adds a specific, non-conflicting constraint and rewrites the question so every updated constraint is explicitly stated in natural language.The process selects a constraint, adds concrete details, and returns an updated constraint list alongside a modified question.Source: Prompt for Constraint Expand · p. 23

Prompt for Conflict Detection

  • Conflict detection checks both whether constraints conflict and whether the question clearly specifies all listed requirements before returning a structured conclusion.The prompt requires explanation before the yes/no conflict conclusion and a separate check of question–constraint coverage.Source: Prompt for Conflict Detection · pp. 24–25

Prompt for Instruction Rewriting (Listing)

  • Listing rewriting modifies the original question according to core requirements without changing its meaning, then emits the result in the specified reply format.The listing prompt frames rewriting as instruction-based data construction while preserving the original question’s meaning.Source: Prompt for Instruction Rewriting… · p. 25

Prompt for Instruction Rewriting (Incorporation)

  • Incorporation rewriting similarly modifies the provided data according to core requirements while preserving the question’s meaning and following the required reply format.Its wording parallels the listing procedure but operates on the supplied data for incorporation-style instructions.Source: Prompt for Instruction Rewriting… · p. 26
Loading 2505.07591v2…