PaperlayerView exact revision on arXiv

Source-linked AI summary

ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions

Xingwei He, Qianru Zhang, Pengfei Chen, Guanhua Chen, Linlin Yu, Yuan Yuan, Siu-Ming Yiu

arXiv:2511.14342v2cs.CL

TL;DR

Prior instruction-following evaluations largely overlook conflicting constraints in complex prompts. ConInstruct benchmarks conflict detection and resolution, finding strong detection but frequent failure to explicitly notify users about conflicts. The benchmark’s scope excludes some complex or ambiguous conflict forms.

  • Problem

    Prior instruction-following work largely assumes coherent constraints, leaving LLM performance on conflicting instructions without systematic evaluation.

    Source: §1 · p. 1

  • Method

    ConInstruct evaluates conflict detection and resolution using instructions with diverse constraints and conflicts within or across constraint types.

    Source: §1 · pp. 1–2

  • Results

    Proprietary LLMs show strong conflict detection, with Claude-4.5-Sonnet achieving the second-highest average F1-score at 87.3%, but models often fail to explicitly notify users about conflicts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight explicit conflict notification and clarification as areas for improvement in instruction-following LLMs.

    Source: §7 · p. 7

    The benchmark may not cover all instruction inconsistencies, since its design avoids overly complex constraints and ambiguous conflicts to enable evaluation.

    Source: A Limitations and Future Work · p. 10

Abstract

from arXiv · show

Instruction-following is a critical capability of Large Language Models (LLMs). While existing works primarily focus on assessing how well LLMs adhere to user instructions, they often overlook scenarios where instructions contain conflicting constraints-a common occurrence in complex prompts. The behavior of LLMs under such conditions remains under-explored. To bridge this gap, we introduce ConInstruct, a benchmark specifically designed to assess LLMs' ability to detect and resolve conflicts within user instructions. Using this dataset, we evaluate LLMs' conflict detection performance and analyze their conflict resolution behavior. Our experiments reveal two key findings: (1) Most proprietary LLMs exhibit strong conflict detection capabilities, whereas among open-source models, only DeepSeek-R1 demonstrates similarly strong performance. DeepSeek-R1 and Claude-4.5-Sonnet achieve the highest average F1-scores at 91.5% and 87.3%, respectively, ranking first and second overall. (2) Despite their strong conflict detection abilities, LLMs rarely explicitly notify users about the conflicts or request clarification when faced with conflicting constraints. These results underscore a critical shortcoming in current LLMs and highlight an important area for future improvement when designing instruction-following LLMs.

1 Introduction

ConInstruct evaluates how LLMs detect and respond to conflicting user constraints, a gap in prior instruction-following evaluations. The results show strong conflict detection but frequent failure to tell users about conflicts.

1 Introduction

  • Prior instruction-following evaluations largely assume coherent constraints, although complex user instructions can contain requirements that cannot be satisfied simultaneously.The paper frames explicit notification and clarification as preferable to silently producing an output that only partially satisfies the instruction.Source: §1 · p. 1
  • ConInstruct covers six tasks and six constraint types, with 7–9 conflict types per instruction, including conflicts within and across constraint types.The paper evaluates conflict detection and analyzes conflict-resolution behavior.Source: §1 · pp. 1–2
  • Proprietary LLMs show strong conflict detection, with Claude-4.5-Sonnet achieving the second-highest average F1-score at 87.3%.Detection ability improves as the number of conflicts in an instruction increases.Source: §1 · p. 2
  • Even strong detectors often generate responses without informing users about conflicts; GPT-4o does so in 97.5% of cases with 1–2 conflicts.Claude-4.5-Sonnet alerts users in 45% of cases, by requesting clarification in 36% or resolving conflicts autonomously in 9%.Source: §1 · p. 2
  • The authors identify explicit conflict communication as an area for future improvement in instruction-following LLMs.Their findings indicate that models often detect conflicts yet fail to convey them explicitly.Source: §1 · p. 2

2 ConInstruct Benchmark

ConInstruct is constructed by expanding seed instructions with diverse constraints, adding directly contradictory constraints, and checking the resulting instructions and conflicts for quality.

2.1 Dataset Construction

  • ConInstruct begins with 100 manually curated seed instructions spanning six NLP tasks and 35 scenario-specific domains.GPT-4o expands each seed instruction by adding all six constraint types: content, keyword, phrase, length, format, and style.Source: §2.1 · p. 2
  • Nine conflict types comprise six within-type conflicts and three between-type conflicts, designed around constraints whose satisfaction can be evaluated.GPT-4o adds a conflict pair for each type: one constraint is taken from the expanded instruction and the other is newly constructed to contradict it.Source: §2.1 · p. 3

2.2 Quality Control

  • Two annotators refine expanded instructions and conflicts, and a third reviews the revisions, removing unreasonable constraints or conflicts.The checks include ensuring all six constraint types and all conflict types are covered, and revising ambiguous conflicts.Source: §2.2 · p. 3

2.3 Dataset Statistics

  • Expanded instructions contain six constraint types and an average of 8.6 conflict types; combining them with conflicts yields 864 instructions with one conflict.The construction process starts with seed instructions, expands them with constraints, then introduces conflicts.Source: §2.1 · p. 2, §2.3 · p. 3, Figure 2 · p. 3

3 Experiment Setup

The experiments evaluate conflict detection and resolution using instructions with either a single conflict or multiple conflicts, across proprietary and open-source models.

3 Experiment Setup

  • The experiment setup evaluates conflict detection and resolution across the prepared instruction sets.The paper describes a common setup for both tasks.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1 Preparing Instructions with Conflicts

  • For single-conflict experiments, each of 100 conflict-free expanded instructions is paired with its distinct conflict types, producing 864 instructions.Nine subsets combine the conflict-free instructions with instructions containing a specific conflict type; the subset sizes vary.Source: §3.1 · p. 4
  • For multiple-conflict experiments, the authors randomly select and shuffle 1–6 conflicts for each seed instruction, producing 100 instructions at each conflict count.Computational constraints limited the construction to one instruction per seed instruction at each count.Source: §3.1 · p. 4

3.2 Evaluation Models

  • The evaluation covers seven proprietary and eleven open-source LLMs, with model outputs limited to 2048 tokens and temperature set to 0.Proprietary models were accessed through official APIs; open-source models were run on A100 GPUs with 40GB of memory.Source: §3.2 · p. 4

4 Experiment Results on Conflict Detection

The section evaluates whether models identify conflicting instructions and reports detection performance for instructions with single and multiple conflicts.

4 Experiment Results on Conflict Detection

  • Conflict detection asks an LLM to classify an instruction as containing conflicts or not.Source: §4 · p. 4

4.1 Instructions with a Single Conflict

  • Claude-4.5-Sonnet (87.3%), Claude-3.5-Sonnet (86.6%), and Claude-3.5-Haiku (85.0%) lead detection; models under 7B underperform random guessing across most conflict types.Source: §4.1 · p. 4, Table 2 · p. 5
  • Claude-3.5-Sonnet scores 92.7% on intra-constraint conflicts and 74.6% on inter-constraint conflicts.Source: §4.1 · p. 4, Table 2 · p. 5

4.2 Instructions with Multiple Conflicts

  • As conflicts increase, larger models generally improve detection, particularly Qwen2.5-[7, 32]B, while sub-7B LLaMA-3.2-3B and Qwen2.5-3B remain lower in recall.Source: §4.2 · p. 4, Figure 3 · p. 5

5 Experiment Results on Conflict Resolution

The section analyzes how LLMs respond to conflicting constraints, whether prompts can steer them toward clarification, and how conflicts affect constraint following. Conflict-aware responses remain limited, and later constraints are more likely to be satisfied than earlier conflicting ones.

5.1 Analysis on Conflict Resolution Behaviors

  • The study classifies responses as unacknowledged, acknowledged with clarification requested, acknowledged and autonomously resolved, or refusals unrelated to conflicts.The authors consider clarification-seeking most desirable because it supports transparency and user control.Source: §5.1 · p. 5
  • GPT-4o labeled 3,600 responses from six LLMs; manual labels on 400 responses agreed with GPT-4o 98% of the time, and Gemini-2.5-Pro showed substantial agreement with it (Cohen’s Kappa 0.746).The study uses Figure 4 to present behavior distributions across different numbers of conflicts.Source: §5.1 · p. 5, Figure 4 · p. 6
  • Conflict-aware Claude responses increase with conflict count, but unacknowledged conflicts remain significant; GPT-4o, DeepSeek-R1, and Qwen2.5-32B predominantly respond without acknowledging conflicts.For Claude-3.5-Sonnet, Behaviors 2 and 3 rise from 32.0% with 1–2 conflicts to 64.0% with 5–6 conflicts.Source: §5.1 · p. 6, Figure 4 · p. 6
  • The authors conclude that safer conflict resolution requires explicitly acknowledging conflicts rather than silently answering with only a subset of constraints satisfied.They describe silent partial satisfaction as potentially misleading to users.Source: §5.1 · pp. 5–6

5.2 Prompting LLMs to Resolve Instruction Conflicts Using Desired Behaviors

  • A guiding prompt induces clarification-seeking behavior but also prompts unnecessary clarification when instructions have no conflicts, so prompting alone does not achieve both desired resolution and accurate execution.Table 3 reports behavior distributions with and without the guiding prompt.Source: §5.2 · p. 6, Table 3 · p. 6

5.3 Analysis on Constraint Priority

  • Among seven LLMs, constraint satisfaction is notably lowest for length constraints and higher for the other five constraint types.The authors evaluate constraint satisfaction on conflict-free instructions before examining the effect of conflicts.Source: §5.3 · p. 6, Figure 5 · p. 7
  • Newly introduced conflicting constraints reduce satisfaction of original constraints, while later-appearing constraints are more likely to be followed, whether original or newly introduced.The analysis focuses on single-conflict instructions and three conflict types: CC, KK, and PP.Source: §5.3 · p. 7, Figure 6 · p. 7

6 Related Work

Related work covers controllable text generation, instruction-following evaluation, and conflict detection across statements, claims, and documents. ConInstruct instead examines how LLMs detect and resolve contradictions among constraints within user instructions.

Controllable Text Generation

  • Controllable text generation studies steer models toward attributes such as sentiment, lexical constraints, and length, and related datasets evaluate or enhance instruction following.The cited examples include Keskar et al. (2019), Dathathri et al. (2020), He (2021), and Zhou et al. (2023a, 2023b).Source: §6 · p. 7

Conflict Detection

  • Unlike these lines of work, ConInstruct evaluates detection and resolution of explicit conflicts among multiple constraints within user instructions.The authors position this as a distinct aspect of instruction-following capabilities.Source: §6 · p. 7
  • Prior conflict-detection work addresses contradictions between statements, claims and evidence, retrieved documents, or parametric and retrieved knowledge, while hallucination detection examines false generated content.The cited domains include natural language inference, fact verification, retrieved-document conflicts, and hallucination detection.Source: §6 · p. 7

7 Conclusion

ConInstruct evaluates LLMs’ ability to detect and resolve conflicting constraints in instructions. The conclusion highlights a gap between strong conflict detection and explicit communication, and calls for models that notify users and seek clarification.

7 Conclusion

  • ConInstruct is a benchmark for evaluating LLMs’ ability to detect and resolve conflicting constraints within instructions.The authors report that proprietary models often generate responses that only partially satisfy constraints without explicitly communicating the conflicts.Source: §7 · p. 7

A Limitations and Future Work

ConInstruct does not cover every form of instruction inconsistency: its constraints prioritize evaluability, and the dataset is primarily text-based. Future work proposes adding implicit and multimodal conflicts.

A Limitations and Future Work

  • The benchmark may omit instruction inconsistencies beyond its covered forms because it prioritizes conflicts whose satisfaction can be evaluated by LLMs or automated programs.This led the authors to avoid overly complex constraints and ambiguous conflicts.Source: A Limitations and Future Work · p. 10
  • The dataset primarily covers text-based instructions, limiting its applicability to conflicts in image-, audio-, or video-based instructions.Source: A Limitations and Future Work · p. 10
  • Future work proposes evaluating implicit conflicts requiring multi-hop reasoning or commonsense inference, as well as conflicts in multimodal settings.Source: A Limitations and Future Work · p. 10

B Data Distribution

This section describes ConInstruct’s task and domain distribution, constraint types and examples, and definitions of conflicts within and across categories, alongside templates for adding constraints and constructing and evaluating prompts for conflict insertion, recognition, instruction following, and resolution.

B Data Distribution

C.1 Content Constraints; C.2 Keyword Constraints

  • Content constraints specify details such as reasons, topics, backgrounds, budgets, and targets, while keyword constraints regulate which terms appear and how they are used.Keyword rules cover inclusion or prohibition, frequency, order, position, part of speech, and definition.Source: §C.1 · p. 10, §C.2 · pp. 10–11

C.3 Phrase Constraints; C.4 Length Constraints

  • Phrase constraints govern phrase inclusion, frequency, and position, while length constraints specify required counts of paragraphs, sentences, or words.Source: §C.3 · p. 11, §C.4 · p. 11

C.5 Format Constraints; C.6 Style Constraints

  • Format constraints cover JSON, Markdown, bullets, language, and case, while style constraints address tone, voice, sentence structure, readability, and perspective.Style examples also include rhetorical choices and author imitation.Source: §C.5 · p. 11, §C.6 · p. 12

1. Conflicts between Content Constraints (CC); 2. Conflicts between Keyword Constraints (KK); 3. Conflicts between Phrase Constraints (PP)

  • Within-category conflicts arise when content requirements contradict, keyword rules oppose one another, or phrase rules demand incompatible wording.Examples include requiring and forbidding a national-park mention, reversing keyword order, and specifying different opening phrases.Source: §1 · p. 12, §2 · p. 12, §3 · p. 13

4. Conflicts between Length Constraints (LL); 5. Conflicts between Format Constraints (FF); 6. Conflicts between Style Constraints (SS)

  • Length conflicts can make paragraph, sentence, and word-count requirements incompatible; format conflicts pit rules such as required bullets against a ban on bullets, and style conflicts set opposing tones or perspectives.Source: §4 · p. 13, §5 · p. 13, §6 · p. 13

7. Conflicts between Keyword Constraints and Phrase Constraints (KP); 8. Conflicts between Phrase Constraints and Content Constraints (PC); 9. Conflicts between Phrase Constraints and Style Constraints (PS)

  • Cross-category conflicts occur when keyword and phrase rules clash, phrase requirements contradict content requirements, or required phrases violate style requirements.Examples include a required phrase containing a forbidden keyword, a required phrase expressing gratitude despite a ban on gratitude, and an informal phrase under a formal-tone rule.Source: §7 · p. 13, §8 · p. 14, §9 · p. 14

E Prompt Templates

  • The prompt template asks models to add constraints from six reference types and apply every sub-constraint without prioritizing one type.Models may select an example template or create their own, provided it satisfies the sub-constraint’s definition.Source: E Prompt Templates · p. 14
  • The seed-expansion prompt adds constraints to a satisfiable instruction, requiring use of all reference constraints while allowing other suitable constraints and paraphrasing.If the seed instruction is a question, it must remain unchanged, with added constraints placed after it; the output is a single-line modified instruction without analysis.Source: E Prompt Templates · pp. 14–15
  • Conflict augmentation requires explicit, unambiguous conflicts across all nine conflict types, using only reference constraints and avoiding internal conflicts among added constraints.The selected constraints must conflict with existing instruction constraints, with one pair for each type, and the examples should span varied scenarios.Source: E Prompt Templates · p. 15
  • The conflict-response evaluator assigns 1 when a model ignores conflicts, 2 when it asks for clarification, 3 when it resolves them itself, and 4 for other responses.The evaluation is based on an instruction containing conflicting constraints and the model response generated from it.Source: E Prompt Templates · p. 15
  • The instruction-following evaluator returns YES only when the response fully satisfies every requirement, and NO when it fails any requirement or gives no relevant information.Its steps extract relevant response text, assess whether constraints are satisfied, then provide the YES/NO judgment.Source: E Prompt Templates · p. 16
  • For two mutually conflicting instructions, the evaluator outputs 1 if Instruction 1 is fully satisfied, 2 if Instruction 2 is fully satisfied, and -1 if neither is fully satisfied.It compares the model response against both instructions, which cannot both be fully satisfied simultaneously.Source: E Prompt Templates · p. 16

F.1 Instruction-Following Evaluation

GPT-4o's automatic evaluations align closely with human judgments across different types of instruction-following constraints, supporting the reliability of the automated assessment.

F.1 Instruction-Following Evaluation

  • GPT-4o’s automatic evaluation aligns closely with human judgment across different types of instruction-following constraints.Source: §F.1 · p. 17, Table 11 · p. 17
  • The authors interpret this alignment as confirming the reliability of GPT-4o’s automatic assessment.Source: §F.1 · p. 17
  • The comparison assesses consistency between GPT-4o and human evaluations across different constraint types.Source: Table 11 · p. 17

F.2 Conflict Resolution Evaluation

Conflict-resolution annotations show that GPT-4o and Qwen2.5-32B predominantly fail to acknowledge conflicts, while Claude-3.5 models partially address them. The authors also report agreement between automated annotators and humans.

F.2 Conflict Resolution Evaluation

  • GPT-4o achieved 98% accuracy against human annotations, and its agreement with Gemini-2.5-Pro had Cohen’s Kappa of 0.746.Source: §F.2 · p. 17, Table 12 · p. 17
  • The analysis used GPT-4o to label a random sample of 400 responses from four models, with 100 responses per model and conflict counts ranging from one to six.Gemini-2.5-Pro and manual labeling on a subset were also used to check annotation quality.Source: §F.2 · p. 17
  • GPT-4o and Qwen2.5-32B predominantly fail to acknowledge conflicts, while Claude-3.5 models partially address them and Claude-3.5-Sonnet most often acknowledges them.The behavior distributions are reported for 400 responses annotated by GPT-4o and by Gemini-2.5-Pro.Source: §F.2 · p. 17, Table 13 · p. 17, Table 14 · p. 17
Loading 2511.14342v2…