Source-linked AI summary
Benchmarking Complex Instruction-Following with Multiple Constraints Composition
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, Minlie Huang
TL;DR
ComplexBench addresses the limited evaluation of complex instruction-following when multiple constraints compose. It introduces a taxonomy, manually collected benchmark, and rule-augmented structure-aware evaluation method; experiments reveal significant deficiencies in current LLMs, especially on complex compositions.
Problem
Existing benchmarks model constraint types but neglect their composition, although composition is an indispensable part of complex instructions.
Method
ComplexBench combines a taxonomy of 4 constraint types, 19 dimensions, and 4 composition types with a manually collected dataset and dependency-based rule-augmented LLM evaluation.
Results
Current LLMs show significant deficiencies on complex instruction-following; GPT-4 still fails 20% of complex instructions, with performance dropping especially on Selection and Chain.
Takeaways & Limitations
ComplexBench provides a systematic way to assess constraint composition and exposes substantial weaknesses in existing LLMs on complex compositions.
Takeaways & Limitations
LLM-based evaluation may be affected by judge bias, high commercial-API costs, and potential data leakage.
Abstract
from arXiv · showhide
Instruction following is one of the fundamental capabilities of large language models (LLMs). As the ability of LLMs is constantly improving, they have been increasingly applied to deal with complex human instructions in real-world scenarios. Therefore, how to evaluate the ability of complex instruction-following of LLMs has become a critical research problem. Existing benchmarks mainly focus on modeling different types of constraints in human instructions while neglecting the composition of different constraints, which is an indispensable constituent in complex instructions. To this end, we propose ComplexBench, a benchmark for comprehensively evaluating the ability of LLMs to follow complex instructions composed of multiple constraints. We propose a hierarchical taxonomy for complex instructions, including 4 constraint types, 19 constraint dimensions, and 4 composition types, and manually collect a high-quality dataset accordingly. To make the evaluation reliable, we augment LLM-based evaluators with rules to effectively verify whether generated texts can satisfy each constraint and composition. Furthermore, we obtain the final evaluation score based on the dependency structure determined by different composition types. ComplexBench identifies significant deficiencies in existing LLMs when dealing with complex instructions with multiple constraints composition.
1 Introduction
ComplexBench addresses the under-evaluation of complex instructions by modeling how multiple constraints compose. It introduces a taxonomy, manually built benchmark, and structure-aware evaluation method for revealing LLM deficiencies.
- Motivation: Existing benchmarks neglect constraint composition, limiting evaluation of complex instruction-following.Complex instructions can combine constraints through coordination, sequential completion, and conditional selection.
- Benchmark: ComplexBench organizes complex instructions into 4 constraint types, 19 constraint dimensions, and 4 composition types.The taxonomy is used to construct a benchmark covering the identified constraint and composition categories.
- Evaluation: ComplexBench evaluates each constraint and composition with yes/no questions answered by LLMs or rules.The method extracts evaluation segments from generated responses before solving the questions.
- Evaluation: Final scores are aggregated through dependencies determined by composition types, while experiments reveal deficiencies across LLMs.The evaluation method explicitly models how composition affects which scoring questions depend on others.
- Benchmark: The benchmark contains a manually collected dataset covering all taxonomy-defined constraint and composition types.The dataset is described as high-quality and designed for complex-instruction following.
2 Related Work
Prior work evaluates instruction following and compositionality in several settings, but ComplexBench targets richer composition in instructions. It extends beyond simple coordination while combining subjective and objective constraints with LLM- and rule-based evaluation.
- Instruction-Following Evaluation: Instruction-following benchmarks evolved from single-constraint tasks toward complex real-world instructions.Earlier work emphasized semantic or format constraints, whereas later benchmarks addressed more complex task formulations.
- Instruction-Following Evaluation: Existing instruction-following benchmarks generally neglect composition among constraints.The paper identifies this omission as distinct from evaluating individual constraint types.
- Compositionality in NLP: Prior compositionality research spans semantic parsing, translation, style transfer, and data-to-text generation.Instruction-following compositionality remains under-explored relative to these traditional NLP tasks.
- Compositionality in NLP: ComplexBench studies Chain, Selection, and nested compositions beyond simple coordination between constraints.This scope distinguishes it from neighboring work on combining multiple control attributes.
3 ComplexBench Framework
ComplexBench organizes complex instructions through a hierarchy of constraint types, dimensions, and composition structures. Its taxonomy covers both basic and nested combinations, with professional instructions exhibiting especially complex compositions.
- Constraint Taxonomy: The framework defines 4 constraint types and 19 dimensions spanning lexical, format, semantic, and utility requirements.Utility constraints include holistic properties such as helpfulness, supportiveness, consistency, and factuality.
- Composition Taxonomy: The taxonomy includes Single, And, Chain, and Selection composition types for structurally combining instruction constraints.Chain orders tasks sequentially, whereas And requires simultaneous satisfaction and Selection chooses a condition-dependent branch.
- Composition Taxonomy: Selection chooses among branches according to a condition, with the expected output determined by the selected branch.The selection function maps the condition to one of m branches and its corresponding expected output.
- Composition Taxonomy: Composition types can be nested, including Selection branches that contain further Selection structures.Tasks in Chain and branches in Selection may themselves contain other composition types.
- Real-World Coverage: Professional instructions contain more complex compositions, including Selection and nested combinations, that current benchmarks have rarely considered.The taxonomy was checked against general and professional instructions collected from an online LLM-based chat service.
4 ComplexBench Construction
ComplexBench is manually constructed from real-world and benchmark instructions through allocation, annotation, validation, and branch-expansion stages. Its evaluation combines rule-based and LLM-based verification, then aggregates results according to composition dependencies.
- Dataset Construction: The dataset construction pipeline comprises reference collection, task allocation, annotation and validation, and Selection branch expansion.The process initially constructs Chinese data and also provides an English version.
- Dataset Construction: Reference instructions come from real-world scenarios and open-source benchmarks, with privacy desensitization and category- and quality-based filtering.Annotators use these references to create new complex instructions rather than constructing every instruction from scratch.
- Dataset Construction: Annotation tasks are partitioned to ensure coverage of required constraint dimensions and composition types.Annotators modify reference instructions to satisfy the requirements of their assigned tasks.
- Evaluation Protocol: Each constraint and composition type receives a manually crafted yes/no scoring question for detailed evaluation.This converts instruction-following assessment into multiple question-answering checks.
- Evaluation Protocol: Rule-verifiable questions use LLM extraction followed by a question-specific rule, while other questions are evaluated directly by an LLM.The hybrid design addresses the complementary strengths of rule-based and LLM-based evaluation.
- Evaluation Protocol: Dependency aggregation propagates failed prerequisite scores through Chain and Selection structures before computing the final DRFR score.Subsequent Chain tasks depend on preceding tasks, and Selection branches depend on choosing the correct branch.
5 Experiments
Experiments show that the proposed evaluation method agrees more closely with human judgments than direct scoring, while current LLMs remain weak on complex constraint compositions.
- Agreement Evaluation: Dependency Aggregation substantially improves overall pairwise agreement over Direct Scoring by modeling composition structures.The comparison uses 500 response pairs sampled from five LLMs.
- Agreement Evaluation: 87.82% overall question-level agreement with humans is achieved by RAL, outperforming Direct Scoring and the no-rule variant.Rule-defined questions comprise 17% of the scoring questions, where rule arguments mainly contribute.
- Main Results: GPT-4 still fails to complete 20% of complex instructions, while performance drops significantly as composition complexity increases, especially for Selection and Chain.Most open-source models also underperform closed-source models on complex composition types.
- Main Results: LLMs generally perform better on Semantic and Utility constraints than on Format and Lexical constraints, while Chain is the hardest composition and Selection ranks second.Selection may be difficult because models must choose the correct branch and execute it without interference from irrelevant branches.
- Decomposition of Instructions: Instruction decomposition does not reliably improve performance: GPT-3.5-Turbo-1106 generally performs worse on decomposed instructions as composition complexity increases.The authors conjecture that cumulative errors across multi-round interactions explain this pattern.
- Coherent Test for Selection: In the coherent test, GPT-4 achieves only 14.9% accuracy on multi-layer nested Selection instructions, exposing weaknesses in following tree-structured instructions.For single-layer Selection, stronger instruction-following ability corresponds to a smaller performance drop in the coherent test.
- Comparisons between Other Capabilities: ComplexBench rankings do not fully match rankings on IFEval, HumanEval, and MATH, indicating that it provides a complementary perspective on LLM evaluation.The discrepancy is especially associated with complex constraint compositions that IFEval does not cover.
6 Conclusion
The paper introduces ComplexBench as a systematic benchmark for complex instruction following and combines structured data construction with structure-aware evaluation. Experiments reveal substantial deficiencies in current LLMs on complex compositions, positioning the benchmark as a complementary evaluation tool.
- Conclusion: ComplexBench organizes complex instructions through a taxonomy of 4 constraint types, 19 constraint dimensions, and 4 composition types.The authors manually collect a corresponding high-quality dataset.
- Conclusion: The benchmark uses structure-aware automatic evaluation and augments LLM-based evaluators with rules to improve evaluation accuracy.The method is designed for constraints composed through different structures.
- Conclusion: Experiments uncover significant deficiencies in current representative LLMs when they handle complex composition types.The authors present ComplexBench as a complementary perspective for LLM evaluation and a source of insights for future improvement.
A Limitations
The paper identifies monolingual construction and LLM-based evaluation as limitations, while noting licensing and responsibility terms for the benchmark and code.
- Monolingual Capability: ComplexBench is primarily constructed from Chinese reference instructions, which may omit complexity elements from other languages and cultures.The authors plan to expand the benchmark to multiple languages in future iterations.
- LLM-based Evaluation: The automatic evaluation method may be affected by LLM-as-Judge biases, including verbosity and self-enhancement.The paper reports generally satisfactory agreement with human judgment but acknowledges potential effects on evaluation correctness.
- LLM-based Evaluation: Using GPT-4-1106 commercial APIs introduces high costs and potential data leakage concerns.The authors identify more accurate and efficient evaluation methods as future work.
- Licensing and Responsibility: ComplexBench is distributed under CC BY 4.0, while its evaluation code is distributed under the MIT license.The paper states that the authors will bear responsibility for rights violations.
C Task Distribution of ComplexBench
ComplexBench organizes instruction tasks using an AlignBench-based taxonomy and defines constraint categories spanning content, format, style, personalization, topic, sentiment, and response quality. Its supplementary materials document annotation and rule-augmented evaluation procedures.
- Task Taxonomy: ComplexBench uses the AlignBench taxonomy to categorize instruction task types and removes mathematics while refining writing ability into four subcategories.The four writing categories are practical, creative, professional, and custom writing.
- Content Constraints: Word Matching and Keywords represent content constraints involving correspondence to instruction keywords and inclusion or exclusion of specified terms.Keywords may be selected from a keyword list.
- Format Constraints: JSON Format, Markdown Format, Bullets Format, Length, and Start with represent response-format and response-position constraints.Length can control words, sentences, paragraphs, bullet points, or keyword counts.
- Semantic Constraints: Language Style, Personalization, Topic, and Sentiment specify stylistic, character, topical, and emotional properties of responses.Sentiment follows six fine-grained categories: Like, Happy, Sad, Disgust, Angry, and Other.
- Quality Constraints and Evaluation: Helpfulness, Target Language, Supportiveness, Consistency, and Factuality capture response quality and language requirements.The supplied task-distribution table is identified as Table 8, while Tables 9–12 document extraction and evaluator prompts and examples.
- Annotation Process: The benchmark annotation process recruited 12 college students, provided a 50-sample training tutorial, used feedback and spot checks, and retrained annotators when issues were found.These procedures were used to calibrate annotation criteria and recheck problematic annotations.
F.1 Guidelines for Data Annotation
The data-annotation guidelines instruct annotators to construct complex instructions, expand selection branches, verify scoring questions, and decompose instructions according to composition types.
- Data Annotation: Annotators construct new complex instructions from reference instructions, task requirements, minimum constraint dimensions, and minimum composition types.The guidelines provide these inputs during annotation.
- Selection Branch Expansion: Selection instructions are expanded by modifying their selection conditions to cover different selection branches.Annotators construct several new instructions from the original selection instructions.
- Agreement Evaluation: For agreement evaluation, annotators compare two model responses and choose whether A is better, tied, or B is better.The comparison uses three preference options.
- Scoring Verification: For scoring-question verification, annotators judge whether a model response satisfies the scoring question associated with its instruction.The judgment uses an instruction, response, and scoring question as inputs.
- Instruction Decomposition: Annotators decompose instructions according to composition types and split the original scoring questions accordingly.Chain is decomposed into sequential tasks, Selection into selection and execution branches, while And remains intact.
G An Example of Data Construction
The data-construction example documents how reference instructions and task requirements support annotation, with translated materials and guidelines covering construction, expansion, evaluation, and decomposition.
- Construction Inputs: The example explains that reference-instruction collection and task allocation produce the table’s Reference Instruction and Task Requirements fields.Each field corresponds to the data-annotation guidelines in Table 13.
- Worked Example: The paper provides a specific data-annotation example in Table 20 to clarify the construction process.The example is presented as an English-translated illustration of data annotation.
- Language Pipeline: The data-construction pipeline first constructs Chinese data from real-world application scenarios, then translates it into English using GLM-4 and manually corrects translation errors.The corrected output forms the English version of ComplexBench.
- Guideline Materials: The benchmark’s annotation materials include English translations of the data-annotation, selection-expansion, preference, scoring-verification, and decomposition guidelines.These materials are presented in Tables 14–19.
I.1 The Influence of Composition Types Nested Methods
Nested multiple Selection is substantially harder than other nested composition types, while adding And has limited impact on overall instruction complexity.
- I.1 The Influence of Composition Types Nested Methods: Nested multiple Selection is significantly more difficult than other nested composition methods.The result is reported using DRFR for GPT-3.5-Turbo-1106 across varying numbers of composition types.
- I.1 The Influence of Composition Types Nested Methods: Adding And has a limited impact on the overall complexity of instructions.
- I.1 The Influence of Composition Types Nested Methods: The findings highlight weaknesses in following complex instructions with multi-layer tree structures.
I.2 Detailed Results of Each Constraint and Composition Type
LLMs handle some constraint dimensions and writing-oriented tasks relatively well, but struggle with length control, logical reasoning, and several precise constraint types.
- I.2 Detailed Results of Each Constraint and Composition Type: 0.532 is the highest accuracy achieved on Length, indicating substantial room for improvement in precise output control.The passage states that even the strongest model reaches only 0.532 accuracy on Length.
- I.2 Detailed Results of Each Constraint and Composition Type: Topic, Markdown Format, Consistency, Sentiment, and Personalization are the easiest constraint dimensions overall.
- I.2 Detailed Results of Each Constraint and Composition Type: Length, Punctuation, Keywords, End with, and Factuality pose the greatest challenges across constraint dimensions.
- I.2 Detailed Results of Each Constraint and Composition Type: LLMs perform relatively better on writing and role-playing tasks but underperform in Logical Reasoning, Advanced Chinese Understanding, and Open-ended Questions.
- I.2 Detailed Results of Each Constraint and Composition Type: All LLMs exhibit significant weaknesses in Logical Reasoning, consistent with reasoning drawbacks reported in AlignBench.