PaperlayerView exact revision on arXiv

Source-linked AI summary

DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents

Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés, Yanis Labrak, Jordi Luque, Milos Cernak, Petr Motlicek

arXiv:2610.21561v1cs.CLcs.AIcs.LG

TL;DR

Procedural hallucination—deviation from conversational process despite plausible utterances—has received less attention than factual errors. DialFork builds a controlled-forking benchmark and evaluates procedural deviation at three granularities; turn-prediction results show a trade-off between sharper non-streaming localization and false alarms, while streaming detectors offer a workflow-free alternative.

  • Problem

    Procedural deviations can occur despite truthful, fluent utterances, but existing hallucination benchmarks assess content truthfulness rather than conversational-process correctness.

    Source: §1 · p. 1

  • Method

    DialFork uses controlled forking to create a 36,278-dialogue benchmark and defines three evaluation tasks for procedural deviation at different granularities.

    Source: §1 · p. 2

  • Results

    With workflow input, non-streaming localization reached MAE 1.09–1.90 (Gemma4-31B Score 85.28), but Gemma4-31B false alarms on conformant dialogues rose to 30.65%.

    Source: §6.3 · p. 9

  • Takeaways & Limitations

    Streaming detectors are lightweight, workflow-free, and flag deviations as they happen, making them a promising basis for real-time guardrails.

    Source: §6.3 · p. 9

    The constructed distribution of language and conversational behavior may differ from dialogues collected from real users and deployed agents.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) deployed as conversational agents are increasingly expected to follow procedural logic. While factual hallucination has received extensive attention, a different failure mode has been overlooked: procedural hallucination, in which an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process. We introduce DialFork, the first benchmark to measure this failure mode on raw dialogue text. DialFork is built by forking conformant dialogues at controlled turn positions and injecting one of four hallucination types, yielding 36,278 dialogues across 24 tasks in 12 domains; 30 human annotators validate the construction. We define three evaluation tasks that characterize procedural deviation at three granularities: how far a set of dialogues deviates from conformant ones (monitoring deployed agents), how far a single dialogue does (procedural retrieval), and at which turn a dialogue first goes off-track (real-time guardrails). For the first two, we propose dialog-flow-aware metrics, FlowKL and FlowScore, that compare dialogues by how they unfold rather than their content. At the dialogue level, FlowScore significantly outperforms all baselines across seven encoders, showing that semantic similarity is no substitute for procedural similarity. We release DialFork under the MIT license at https://github.com/idiap/dialfork, hoping the community will measure conversational process alongside content.

1 Introduction

Conversational agents are expected to follow structured procedures, but existing hallucination benchmarks focus on factual content rather than whether a dialogue follows its intended process. DialFork addresses this gap with controlled dialogue forking to study procedural deviations.

1 Introduction

  • Agents deployed in roles such as customer support, scheduling, troubleshooting, and clinical intake are expected to follow structured steps toward a completed goal.The expectation applies beyond classical task-oriented dialogue wherever conversational process matters alongside utterance content.Source: §1 · p. 1
  • Procedural hallucination is deviation from conversational process, including skipped steps or changed goals, even when individual utterances remain plausible or truthful.The paper distinguishes this failure from factual hallucination and says standard content-truthfulness benchmarks are blind to it.Source: §1 · p. 1
  • DialFork simulates procedural hallucinations by forking conformant dialogues at controlled turns and injecting a hallucination instruction into the system agent’s context.The shared prefix and progressively earlier fork positions establish a procedural-similarity ordering distinct from semantic similarity.Source: §1 · p. 1, Figure 1 · p. 2
  • The benchmark spans four hallucination types and supports three evaluation granularities, with FlowKL and FlowScore comparing dialogue flow rather than content.The evaluation tasks address deviation across a set of dialogues, within one dialogue, and at the turn where deviation begins.Source: §1 · p. 2

2 Related Work

Prior hallucination benchmarks assess what dialogue says, while procedural-adherence and dialogue-similarity research typically uses annotations or text-level similarity. DialFork instead evaluates deviation from procedure directly on raw dialogue text.

2 Related Work

  • Hallucination research and dialogue-level benchmarks focus on factuality or content faithfulness, not whether a conversation unfolds correctly.The related work describes dialogue benchmarks such as DiaHalu as extending content-oriented categories to multi-turn conversations.Source: §2 · p. 2
  • Procedural-adherence studies typically evaluate annotated actions, intents, or slots, while embedding-based dialogue similarity collapses turns and discards their order.TaskDiff uses annotated intents and slots; DIALFORK instead measures procedural deviation directly from raw dialogue text, regardless of whether an LLM or human produced it.Source: §2 · p. 2

3 Procedural Hallucination

The paper models intended conversational logic as a workflow and defines procedural hallucinations as corruptions to different parts of that workflow. This abstraction is intended to apply across goal-directed conversational agents.

3.1 Formal Model

  • The workflow model W = (S, A, δ, O) represents conversational states, agent actions, user-input-conditioned transitions, and the agent’s objective.The model abstracts procedural logic that is often implicit in instructions, business processes, or conversational norms.Source: §3.1 · p. 2
  • The workflow abstraction is intended for goal-directed agents, including task-oriented dialogue, medical intake, and customer support.Examples pair a dialogue policy, clinical protocol, or troubleshooting procedure with the corresponding task-completion objective.Source: §3.1 · p. 2

3.2 Hallucination Types

  • The four hallucination types corrupt distinct workflow components: identity corruption replaces the workflow, policy corruption changes transitions, goal replacement changes the objective, and conflicting constraints add a competing objective.Policy corruption can repeat, skip, loop, or randomize steps; conflicting constraints can intermittently override the original objective.Source: §3.2 · p. 3

4 The DIALFORK Dataset

DialFork uses STAR’s workflows and role-play instructions to generate conformant dialogues, then forks them with controlled procedural deviations. Its resulting dataset is large, and human judgments support the intended fork ordering and behavior.

4.1 Construction Pipeline

  • STAR supplies workflows and the original role-play instructions, which the generation pipeline gives to LLM agents in place of the human participants.The corpus releases both workflow ground truth and the instructions used to produce its dialogues; DIALFORK uses the instructions and workflows, not STAR’s original dialogue text.Source: §4.1 · p. 3
  • Seeded replay reproduces the conformant prefix before injecting a sampled hallucination instruction at one of four evenly spaced system-turn fork positions.Earlier forks share less of the conformant dialogue and produce more deeply deviant continuations.Source: §4.1 · p. 4
  • Human validation found blind conformance ratings rising from 14/100 at fork position k=1 to 58 at k=4, versus 94 for conformant dialogues.Annotators confirmed the injected type in 91% of forks, located its onset within one turn of the fork in 77%, and judged the pre-fork turns workflow-conformant in 95%.Source: §4.1 · p. 4

4.2 Dataset Statistics

  • 36,278 dialogues were constructed from 2,134 conformant dialogues, with 16 forked variants per source dialogue; forked dialogues averaged 21.18 turns versus 15.52 for conformant ones.As fork position advances, the intended-workflow prefix grows from 0 to 12.5 turns while the corrupted continuation shrinks from 20.9 to 8.0 turns.Source: §4.2 · pp. 4–5

5 Evaluation Tasks and Methods

The three tasks measure procedural deviation at set, dialogue, and turn level, using raw dialogue text to assess how conversations unfold rather than what they say. The methods compare reference and target dialogue sets, rank individual dialogues by flow similarity, and locate the first off-track turn.

5.1 Task 1: Set-to-Set Distance Ranking

  • FlowKL compares reference and target sets by how well their dialogues fit a human-readable graph of conformant workflow transitions.It maps turns to graph nodes, scores dialogue transition likelihoods, then compares the distributions of those scores.Source: §5.1 · p. 6

5.2 Task 2: Dialog-to-Dialog Distance Ranking

  • Task 2 ranks four forked dialogues by distance from each conformant dialogue, evaluating methods with mean Kendall’s τ across conformant dialogues.The baselines pool turn or full-dialogue embeddings; FLOWDTW aligns turn sequences, while FLOWSCORE uses soft turn-level matching without strict monotonic warping.Source: §5.2 · pp. 6–7

5.3 Task 3: Hallucination Turn Prediction

  • Task 3 predicts the first hallucinated system turn, or none for a conformant dialogue, across streaming and two non-streaming regimes.Streaming uses conformant references without workflow access; non-streaming prompting uses the full dialogue, with workflow access only in the privileged third regime.Source: §5.3 · p. 7
  • Streaming detectors flag either a transition below a conformant-calibrated threshold or a surprisal jump above its conformant-calibrated threshold.The non-streaming regimes use zero-shot LLM prompting; evaluation combines turn-localization error with coverage measures.Source: §5.3 · p. 8

6 Experiments

Across the three tasks, dialog-flow-aware methods recover procedural deviations, with FLOWSCORE strongest for ranking individual dialogues across encoders. Streaming detectors localize deviations with few false alarms, while non-streaming LLMs localize more sharply but abstain more often or raise more false alarms.

6.1 Task 1: Set-to-Set Distance Ranking

  • Set-to-set metrics recover the expected fork ordering with average τ ≥0.88, while length-only controls do not; FLOWKL performs best on SSPC.The three metrics do not differ significantly overall, and FLOWKL performs on par with FBD and PRD.Source: §6.1 · p. 8

6.2 Task 2: Dialog-to-Dialog Distance Ranking

  • Across seven encoders, FLOWSCORE is strongest with significant gains over every method, lifting all above τ=0.81; Full ranks forks at τ=0.519–0.747.FLOWSCORE recovers the ordering at τ=0.812–0.845, showing semantic similarity alone does not capture how dialogues unfold.Source: §6.2 · p. 9
  • Turn-count difference is near chance (τ=0.126); FLOWSCORE beats shared-prefix ranking (τ=0.682) for every encoder (p < 0.001).Shared prefixes expose fork position from the construction, but replayed prefixes can contain minor rewordings.Source: §6.2 · p. 9

6.3 Task 3: Hallucination Turn Prediction

  • Streaming detectors localize forks within ∼4.6–5.0 turns at ∼3.5–5.6% false alarms; IT+NC achieves the best Score (71.07).NC weighting improves both detectors at the cost of slightly more false alarms.Source: §6.3 · p. 9
  • Gemma4-31B reaches MAE 3.02 vs. 4.61 and Score 82.82, but LLMs flag only 72–84% of forks versus 93.82% for streaming.Providing the workflow sharpens localization further (MAE 1.09–1.90; Gemma4-31B: Score 85.28), but false alarms reach 30.65% for Gemma4-31B and up to 82.80%.Source: §6.3 · p. 9

7 Conclusions

The paper introduces DIALFORK to make procedural hallucination measurable from raw dialogue text, identifying procedural similarity as distinct from semantic similarity. Its three tasks support monitoring, retrieval, and real-time guardrail research.

7 Conclusions

  • DIALFORK makes procedural hallucination systematically measurable from raw dialogue text, exposing procedural similarity as an axis distinct from semantic similarity.The authors propose its three tasks as a basis for distributional monitoring, procedural retrieval, and real-time guardrails.Source: §7 · p. 9

Limitations

The evidence is bounded by the benchmark's corpus, constructed deviations, and baseline methods. Each limits how directly reported results characterize real-world dialogue failures or the strongest possible detectors.

Limitations

  • Empirical results reflect STAR's task and domain distribution, although the pipeline and formal model apply to any corpus with workflow-level metadata.Extending the benchmark to other corpora, including non-English dialogues or workflows and domains beyond STAR's, is left to future work and users.Source: Limitations · p. 10
  • Each fork contains one hallucination type at one point, leaving compound or cascading failures unevaluated and the four-type taxonomy potentially incomplete.The authors note that real failures can involve multiple co-occurring types or one corruption inducing another.Source: Limitations · p. 10
  • Because dialogues are LLM-generated role-play, methods that work on DIALFORK may behave differently on naturally occurring conversations.The human study confirms intended deviations across five tasks, but does not establish detector performance on real-user dialogues.Source: Limitations · p. 10
  • Reported methods are initial baselines, not the strongest possible detectors; methods using generator-internal signals are excluded by design.Streaming uses simple N-gram models, while non-streaming uses zero-shot prompting without fine-tuning, few-shot examples, or chain-of-thought elicitation.Source: Limitations · p. 10

8 Ethical Considerations

DIALFORK uses synthetic dialogues and volunteer judgments rather than real user data. The authors caution that its controlled results are not direct estimates of deployed-system behavior and recommend domain-specific validation before high-stakes use.

8 Ethical Considerations

  • The benchmark contains no real user data; volunteers consented, could stop at any time, and were identified only by opaque IDs.Participants judged synthetic dialogues and chose whether to be acknowledged by name or remain anonymous.Source: §8 · p. 10
  • The authors caution against deploying DIALFORK-validated detectors as high-stakes guardrails without domain-specific validation.The 24 service-oriented tasks may not represent real-world settings, where missed deviations or excessive false alarms carry real costs.Source: §8 · p. 11
  • Claude assisted with paper writing and editing and annotation-tool development; the authors reviewed and verified the AI-assisted text and code.The authors state that they take full responsibility for the content.Source: §8 · p. 11

A Detailed Per-Task Dataset Statistics

This section details per-task dataset statistics and construction, including dialogue-length variation, STAR metadata, and controlled orchestration for generating conformant and forked dialogues. It also documents injection and detection prompts, illustrates how forked dialogues diverge, and defines Task 3’s composite score.

SSPC Variants

A Detailed Per-Task Dataset Statistics

GR Variants

  • GR variants replace task progress with information collection, education, alternatives, registration, summaries, or feedback gathering.Source: GR Variants · p. 14

CSC Variants

  • CSC variants impose persistent questioning, conversation extension, excessive confirmation or information requests, related topics, or explanations before answers.Source: CSC Variants · p. 14

C STAR Workflow Example; D STAR Role-Play Metadata Example

E Generation Details

  • Conformant agents follow workflow exemplars while users reveal information gradually; a shared per-task turn budget bounds conformant dialogues and forks, though forks remain longer on average.Source: E Generation Details · p. 16

E Generation Details; F Hallucination Injection Prompts

F Hallucination Injection Prompts

  • The SSPC, GR, and CSC injection listings use sampled placeholders to specify corrupted policies or goals and their expected behavioral effects.The SSPC template directs the system to disregard the prescribed workflow step order.Source: F Hallucination Injection Prompts · p. 18

G Qualitative Dialogue Examples

  • Forked variants share a dialogue ID and prefix with the conformant dialogue, then diverge from the fork turn according to one of four hallucination types.The apartment-scheduling example shows policy corruption jumping to the workflow’s last step, goal replacement suggesting alternatives, and constraint corruption repeatedly confirming information.Source: G Qualitative Dialogue Examples · pp. 18–20

J Statistical Testing Protocol

The protocol tests ranking metrics against chance and compares FLOWKL with baselines across 96 task–type units; it also specifies prompt regimes and encoders for turn detection.

J Statistical Testing Protocol

  • Task 1 tests Kendall’s τ against chance per task–type unit with one-sided permutation tests, combines p-values using Fisher’s method, and tests exact rankings against 1/4!.The protocol uses 96 units and a one-sided binomial test for exact four-way rankings.Source: J Statistical Testing Protocol · pp. 20–21

K Encoder Models

  • The dialog-only prompt judges the first inconsistent system turn from context and goals; the workflow prompt instead flags a turn whose state is unreachable from the current workflow position.Both return a 1-based system-turn index, or null when no deviation is found.Source: J Statistical Testing Protocol · p. 21, K Encoder Models · p. 22

J Statistical Testing Protocol

  • Task 2 compares FLOWScore with each baseline by one-sided paired Wilcoxon tests per encoder across 8,536 dialogue–fork units, with Holm correction and 95% paired-bootstrap intervals.Intervals use 10,000 resamples.Source: J Statistical Testing Protocol · p. 21
  • FLOWKL recovered exact four-way rankings in 67.7% of units, above 4.2% chance; FBD reached 77.1% and PRD 68.8%, while Turn-count diff. was not above chance.Word-count diff. was only weakly above chance (p = 0.048; exact recovery in 15.6% of units).Source: J Statistical Testing Protocol · p. 21
  • FLOWKL’s mean Kendall’s τ difference was −0.031 vs. FBD and +0.003 vs. PRD; neither paired comparison was significant.Holm-corrected Wilcoxon p-values were 0.43 and 0.79, respectively.Source: J Statistical Testing Protocol · p. 21
  • FLOWKL exceeded Turn-count diff. by +0.969 and Word-count diff. by +0.681 in mean Kendall’s τ; both comparisons had p < 0.001.The reported 95% confidence intervals were [0.835, 1.104] and [0.559, 0.809], respectively.Source: J Statistical Testing Protocol · p. 21

K Encoder Models

  • Tasks 2 and 3 use SBERT, LaBSE, BGE-M3, Qwen3-Embedding, EmbeddingGemma, DSE, and D2F encoders.D2F is also used in FLOWKL.Source: K Encoder Models · p. 21

L Task 2: Significance Tests

Table 7 presents paired FlowScore comparisons and specifies the unit used for Task 2.

L Task 2: Significance Tests

M Streaming Methods: Encoder Ablation

In streaming detection, encoder choice strongly shapes performance, while node-confidence weighting trades lower hallucinated MAE for mixed Score effects. The human study validates DialFork’s fork labels through staged, blinded judgments and analyses of fork order, injection, onset, and prefix conformance.

M Streaming Methods: Encoder Ablation

N.1 Research Questions

  • The study tests whether fork position tracks perceived deviation, injected behavior appears, onset matches the fork turn, and prefixes follow the intended workflow.These four research questions validate the corresponding benchmark labels and claims.Source: §N.1 · p. 26, N Human Evaluation Study · p. 26

N.3 Sample

  • The final sample comprised 428 conversations, above the 380 required: 342 forked dialogues (85 or 86 per hallucination type and fork position) and 86 conformant examples.They were assigned in five-conversation miniblocks (four forked, one conformant), preserving balance by hallucination type and fork position even if annotators stopped early; no annotator saw two versions of a source dialogue.Source: §N.3 · pp. 26–27

N.4 Annotation Protocol

  • Annotators first rated conformance without construction cues, then checked the injected instruction and onset, and finally judged the pre-fork prefix.The workflow stayed visible; server-side blinding withheld cues and fork turns until their respective steps, and foil instructions checked indiscriminate confirmation.Source: §N.4 · pp. 27–28, §N.5 · p. 28

N.6 Analysis

  • The analysis measures fork-order association, confirmation against foils, onset error, prefix conformance, and inter-annotator reliability.Fork-order statistics are computed within annotator–task pairs; agreement uses AC1 for categorical answers and Krippendorff’s α for continuous answers.Source: §N.6 · pp. 28–29

N.7 Results

Human judgments generally follow the benchmark’s intended fork-order, injection, localization, and prefix labels. Agreement is substantial for injected behavior and high for onset, while embedding-based dialogue distances track perceived deviation more closely than length-only controls.

RQ1: fork order

  • 14/100 at k=1 rising to 58 at k=4: perceived conformance increased with fork position for WIC, SSPC, and GR, while conformant dialogues scored 94.A later fork was rated above an earlier one in 80% of pairs; the mean within-annotator τb was 0.57.Source: §N.7 · p. 30
  • CSC forks were rated below conformant dialogues in 87% of pairs, but perceived deviation changed little across positions, from 59 at k=1 to 70 at k=4.Including CSC, conformance rose from 26 at k=1 to 61 at k=4, and later forks were rated above earlier ones in 72% of pairs.Source: §N.7 · pp. 30–31

RQ2: injection validity

  • 91% of forked dialogues had the true injected instruction confirmed, versus 8% of conformant dialogues with foils; forked-dialogue foils were confirmed in 45%.True instructions were confirmed twice as often as foils on forked dialogues.Source: §N.7 · p. 31

RQ3: localization

  • 67% of confirmed forks had onset placed exactly at the fork turn, and 77% were within one turn; 96% of onsets were at or after it.Shared next steps can delay visible deviation by a turn or two.Source: §N.7 · p. 31

RQ4: prefix conformance

  • For forks with a pre-fork segment, annotators judged the assistant followed the workflow in 86% of cases and mostly followed it in another 9%.The combined 95% share supports seeded replay generating prefixes under the intended policy.Source: §N.7 · p. 31

Agreement and reliability

  • Agreement was AC1 = 0.69 for whether injected behavior occurred, AC1 = 0.85 for prefix judgments, and Krippendorff’s α = 0.84 for onset turns.Excluding one binary slider user changed mean ratings per fork position by at most 3 points and mean τb from 0.41 to 0.43.Source: §N.7 · p. 31

Relation to automatic metrics

  • Embedding-based Task-2 distances correlated with perceived deviation at Spearman ρ = 0.58–0.67, versus 0.36–0.37 for length-only controls.The pooled correlation reflects differences across sources, tasks, and hallucination types, so it does not rank Task-2 methods.Source: §N.7 · p. 31

Related papers

Loading 2610.21561v1…