PaperlayerView exact revision on arXiv

Source-linked AI summary

StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following

Jinnan Li, Jinzhe Li, Yue Wang, Yi Chang, Yuan Wu

arXiv:2502.14494v2cs.CL

TL;DR

Existing evaluations emphasize fine-grained constraint satisfaction but lack systematic treatment of structural dependencies between dialogue turns. StructFlowBench combines a structural-flow taxonomy with constraint-based evaluation and evaluates 13 LLMs, finding variation in their structural processing capabilities.

  • Problem

    Existing evaluations focus on fine-grained constraints yet lack a systematic framework for inter-turn structure, despite its role in reflecting user intent and distinguishing multi-turn interaction.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    StructFlowBench combines intra-turn and structural constraints with a six-category flow taxonomy used to evaluate models and guide scenario-specific dialogue generation.

    Source: §1 · p. 2, Parameter Setting · p. 4

  • Results

    DeepSeek-v3 outperforms the other evaluated models across all metrics, while mid-tier models show greater instability, particularly in ISR and WCSR.

    Source: Overall Results · p. 6

  • Takeaways & Limitations

    The benchmark provides a framework for assessing multi-turn dialogue coherence, goal clarity, and transition naturalness and is intended to guide future instruction-following advances.

    Source: §5 · p. 9

    The benchmark explores evaluation directions rather than deliberately increasing difficulty, so leading models tend to reach near-maximum scores on many metrics.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multi-turn instruction following capability constitutes a core competency of large language models (LLMs) in real-world applications. Existing evaluation benchmarks predominantly focus on fine-grained constraint satisfaction and domain-specific capability assessment, yet overlook the crucial structural dependencies between dialogue turns that distinguish multi-turn from single-turn interactions. These structural dependencies not only reflect user intent but also establish an essential second dimension for the instruction following evaluation beyond constraint satisfaction. To address this gap, we propose StructFlowBench, a multi-turn instruction following benchmark with structural flow modeling. The benchmark defines an innovative structural flow framework with six fundamental inter-turn relationships. These relationships introduce novel structural constraints for model evaluation and also serve as generation parameters for creating customized dialogue flows tailored to specific scenarios. Adopting established LLM-based automatic evaluation methodologies, we conduct systematic evaluations of 13 leading open-source and closed-source LLMs. Experimental results reveal significant deficiencies in current models' comprehension of multi-turn dialogue structures. The code is available at https://github.com/MLGroupJLU/StructFlowBench.

1 Introduction

StructFlowBench addresses evaluations that treat dialogue turns as concatenated single-turn interactions and lack a systematic account of structural flow. It introduces a benchmark combining structural constraints with a six-category taxonomy for analyzing and generating multi-turn dialogue flows.

1 Introduction

  • Existing evaluations neglect inter-turn dependencies, limiting their ability to capture complex dialogue coherence, user goals, and natural transitions.They emphasize individual constraint compliance without a systematic framework for dialogue structural flow.Source: §1 · p. 1
  • The benchmark combines intra-turn and inter-turn constraint evaluation to assess both individual constraint satisfaction and coherence across turns.It integrates five structural constraints with eight intra-turn instruction constraints.Source: §1 · p. 2
  • The six-category taxonomy covers Follow-up, Refinement, Recall, Summary, Expansion, and Unrelatedness relationships between dialogue turns.It supports structural diagnosis, implicit-intent inference, and controlled generation of dialogues with predefined flow patterns.Source: §1 · p. 2
  • StructFlowBench is presented as a structurally annotated benchmark using structure-driven generation to simulate complex dialogue scenarios.The paper also reports systematic evaluation of 13 state-of-the-art models, including three closed-source and ten open-source LLMs.Source: §1 · p. 2

2 Related Work

Prior benchmarks address multi-turn dialogue evaluation and fine-grained instruction following, but the related work identifies gaps in systematic structural modeling and real-world user intentionality.

2.1 Benchmarks for Multi-Turn Dialogues

  • Multi-turn benchmarks assess dialogue capabilities, while most do not prioritize instruction following or systematic structural information.MT-Eval explores four structures but lacks a systematic framework integrating structural aspects for comprehensive evaluation.Source: §2.1 · p. 3

2.2 Benchmarks for Instruction Following

  • Instruction-following benchmarks commonly use constraints, but prior work largely focuses on single-turn interactions that incompletely capture users’ intentionality and goal orientation.ComplexBench studies constraint composition in single-turn dialogues; some work expands instructions into multi-turn form without fully modeling real-world user goals.Source: §2.2 · p. 3

3 StructFlowBench

StructFlowBench defines structural flow categories and constraint types, then uses configurable dialogue generation and constraint annotation to construct its benchmark.

3.1 Structural Flow Taxonomy

  • The taxonomy identifies six dialogue patterns from existing LLM and human multi-turn datasets and uses them to analyze and construct structural flows.The patterns are Follow-up, Refinement, Recall, Expansion, Summary, and Unrelatedness.Source: §3.1 · p. 3
  • The six patterns distinguish adjacent-turn continuation or revision, long-range reference, topic branching or consolidation, and independent new topics.Follow-up builds on the previous turn; Refinement modifies its prompt; Recall reaches back two or more turns; Expansion fans out; Summary fans in; Unrelatedness introduces an independent prompt.Source: §3.1 · pp. 3–4

3.2 Constraint Categories

  • The constraint scheme separates eight intra-turn types from five structural types designed to maintain cross-turn coherence and continuity.Structural constraints address follow-ups, refinements, recalls, expansions, and summaries; Unrelatedness is excluded.Source: §3.2 · p. 4

3.3 Data Construction Pipeline

  • The construction pipeline has three components: parameter setting, two-step dialogue generation, and constraint extraction and addition.The pipeline converts selected structural flows into dialogue data and associated constraints.Source: §3.3 · p. 4, Figure 3 · p. 4

Parameter Setting

  • Generation parameters include task, topic, user characteristics, and structural flow, with eight task types, 22 topics, and 14 manually designed templates.The templates draw on WildChat and cover most pairwise combinations of the six structural relations.Source: Parameter Setting · p. 4

Two-Step Dialogue Generation

  • GPT-4o first generates an intermediate dialogue plan from the structural-flow template, then generates the complete dialogue from that plan.Locally deployed mini-models screen plans, followed by manual inspection of error data.Source: Two-Step Dialogue Generation · p. 5

Constraint Extraction and Addition

  • GPT-4o extracts intra-turn constraints for manual validation, after which structural constraints are assigned to dialogue turns from their flow information.The construction pipeline depicts these as the final annotation steps.Source: Figure 3 · p. 4, Constraint Extraction and Addition · p. 5

3.4 Benchmark Dataset Statistics

This section reports StructFlowBench’s dataset scale and describes the response-evaluation criteria and metrics, including a weighted measure that assigns greater importance to structural constraints.

3.4 Benchmark Dataset Statistics

  • The benchmark contains 155 multi-turn dialogues, 643 turns, and 1,775 constraints across 8 task types, 22 topics, and 13 constraint types.Table 1 compares related datasets by fine-grained constraints, multi-turn dialogue assessment, and structural information.Source: §3.4 · p. 5

Evaluation Criteria

  • The evaluation uses curated dialogue histories as Golden Contexts rather than model-generated contexts to support consistent assessment across models.The stated aim is to reduce bias and noise and improve reliability, fairness, and comparability.Source: Evaluation Criteria · p. 5
  • Instructions are decomposed into independent constraints, each assessed with a binary Yes-or-No question and aggregated into a checklist.The checklist is intended to cover all critical instruction constraints.Source: Evaluation Criteria · p. 5
  • GPT-4o evaluates responses using the Golden Context, test-model response, constraint checklist, and a prompt template.The evaluation builds on the LLM-as-evaluator approach described for MT-Bench.Source: Evaluation Criteria · p. 5

Evaluation Metrics

  • CSR, ISR, and DRFR are existing measures; WCSR addresses CSR’s equal constraint weighting and ISR’s all-or-nothing scoring by weighting constraint types.In this framework, intra-turn constraints have weight 1 and structural constraints weight 2; the authors say WCSR provides a more nuanced evaluation.Source: Evaluation Metrics · p. 6

4 Experiments

Across 13 models, DeepSeek-v3 leads overall, but refinement and format adherence remain difficult; StructFlowBench also scores highest among five datasets on complex-scenario suitability. Human review shows GPT-4o evaluations align with experts at approximately 0.75 Kappa.

4.1 Experimental Setup

  • The evaluation covers 13 models: three closed-source and ten open-source systems.The paper reports results across four key metrics and structural constraints, with detailed breakdowns by intra-turn constraints and task types in Appendix B.Source: §4.1 · p. 6, Overall Results · p. 6

4.2 Main Results

  • DeepSeek-v3 outperforms all other models across all metrics, while mid-tier models show instability and Deepseek-R1-Distill-Qwen-7B and Mistral-7B-Instruct-v0.3 are weakest.DeepSeek-v3 is open-source and surpasses its closed-source counterparts; Deepseek-R1-Distill-Llama-8B also outperforms its Llama-3.1-8B-Instruct source across metrics.Source: Overall Results · pp. 6–7, Table 2 · p. 7

Structural-Constraint-Categorized Performance

  • Follow-up and recall structures are handled well, but refinement is most challenging, with even DeepSeek-v3 scoring only 0.8 in that category.The analysis attributes refinement errors to models mistaking refinement for follow-up and forgetting earlier constraints that should remain intact.Source: Structural-Constraint-Categorized… · p. 7, §4.3.2 · p. 9

Intra-Turn-Constraint-Categorized Performance

  • Format-related constraints remain difficult even for top-performing models, despite strong performance on rule-based constraints and near-perfect satisfaction rates by DeepSeek-v3, Gemini-1.5-Pro, and GPT-4o.The evaluated format constraints include basic, template, and quantity formats.Source: Intra-Turn-Constraint-Categorized… · p. 7

Task-Categorized Performance

  • DeepSeek-v3 remains best overall across task types but leads only four, while Gemini-1.5-Pro, Claude-3.5-Sonnet, and GPT-4o lead in other task categories.GLM-4-9B-Chat and both Qwen2.5 instruction models are described as consistent, cost-effective alternatives; Mistral underperforms across nearly all tasks.Source: Task-Categorized Performance · p. 8

4.3.1 Complex Scenario Suitability Study

  • StructFlowBench scores highest across logical coherence, goal clarity, and transition naturalness, with a confusion factor of 0.83.MT-Bench-101 ranks closely behind, whereas WILDCHAT performs worst across the three dimensions.Source: §4.3.1 · p. 8

4.3.2 Fail Case Analysis of Refinement

  • Refinement failures arise when models confuse refinement with follow-up and forget earlier constraints that should remain unchanged.The evaluation compared five datasets using 50 randomly selected English multi-turn dialogues per dataset, GPT-4o scoring, and three properties rated from 1 to 5.Source: §4.3.2 · pp. 8–9, §4.3.1 · p. 8

4.4 Human Verification

  • GPT-4o evaluations of 30 Qwen2.5-7B-Instruct dialogues achieve approximately 0.75 Kappa agreement with two domain experts.Experts assessed constraint adherence using binary Yes/No judgments.Source: §4.4 · p. 9

5 Conclusion

StructFlowBench combines a dual-constraint evaluation system with a six-category structural-flow taxonomy to assess complex multi-turn dialogue. Evaluations of 13 LLMs identify structural-processing capabilities and offer guidance for future instruction-following advances.

5 Conclusion

  • StructFlowBench combines dual-constraint evaluation and a six-category structural-flow taxonomy to assess complex dialogue and evaluate 13 representative LLMs.The framework targets logical coherence, goal clarity, and transition naturalness across multi-turn dialogues.Source: §5 · p. 9

Limitations

StructFlowBench explores a new direction in multi-turn instruction-following evaluation rather than targeting a harder evaluation set. Its single-linear-relationship design omits coexisting dialogue dependencies, limiting comprehensive modeling of hierarchical structure.

Limitations

  • The benchmark is an exploration of a new evaluation direction, not a targeted effort to increase evaluation difficulty.Source: Limitations · p. 9
  • Future difficulty enhancements could extend dialogue turns, add per-turn constraints, and include complex reasoning or retrieval-augmented generation tasks.Source: Limitations · p. 9
  • StructFlowBench uses a single linear relationship for structural flow, retaining one link when a turn has multiple dependencies and potentially limiting hierarchical dialogue modeling.For example, a third turn linked to the first by recall and the second by follow-up retains only the recall relationship.Source: Limitations · p. 10

Ethics Statement

The dataset uses GPT-4o for dialogue generation and constraint annotation, followed by manual review. The authors note residual bias and error risks, possible benchmark-validity effects from public release, and recommend cautious, complementary evaluation.

Ethics Statement

  • GPT-4o generates dialogues and annotates constraints, while manual review filters inappropriate content.Source: Ethics Statement · p. 10
  • Generation biases and human-review oversights may leave residual dataset errors or biases that cannot be completely eliminated.Source: Ethics Statement · p. 10
  • Public availability risks misuse for training and compromised benchmark validity, so the authors recommend caution and complementary evaluation methods.Source: Ethics Statement · p. 10

A Details of Topics and Tasks

StructFlowBench spans 22 topics and comprises seven NLP tasks plus one mixed-category task; Table 3 is identified as detailing the task distribution.

A Details of Topics and Tasks

B Detailed Results Categorized by Intra-turn Constraints and Task Types

The appendix points readers to performance results organized by intra-turn constraints and task categories, with a radar chart comparing both perspectives.

B Detailed Results Categorized by Intra-turn Constraints and Task Types

C Details of Prompts

This appendix collects the study’s prompt templates and reference materials alongside tables describing StructFlowBench’s constraint system, constraint distribution, and model performance.

C Details of Prompts

  • Figures 7–10 present templates for dialogue-plan generation, complete dialogue generation, constraint extraction, and GPT-4o evaluation.Figure 11 lists task requirements for human evaluators.Source: C Details of Prompts · p. 12, Figure 7 · p. 17, Figure 8 · p. 18
  • Table 5 describes the distribution of structural constraints Follow-up, Refinement, Expansion, Summary, and Recall across constraint types C1–C8.Table 4 is titled as the benchmark’s constraint system.Source: Table 4 · p. 13, Table 5 · p. 14
  • Tables 6 and 7 report model performance by intra-turn constraint and task, respectively.Figure 6 presents radar charts for intra-turn-constraint-categorized and task-categorized performance.Source: Table 6 · p. 14, Table 7 · p. 14, Figure 6 · p. 14
  • Table 8 provides an example of synthetic data.Source: Table 8 · p. 15
  • Table 9 overviews selected large language models, their parameter sizes, and reference links.Source: Table 9 · p. 16

Related papers

Loading 2502.14494v2…