PaperlayerView exact revision on arXiv

Source-linked AI summary

Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction

Sergio Burdisso, Srikanth Madikeri, Petr Motlicek

arXiv:2410.18481v2cs.CLcs.AIcs.LG

TL;DR

Automatic extraction of structured workflows from dialogs is underexplored, despite its potential uses in dialog-system design and domain-specific LLM grounding. Dialog2Flow learns utterance embeddings grouped by action function using a semantic soft contrastive loss and a unified corpus of twenty task-oriented dialog datasets. The authors report that D2F variants outperform embedding baselines, and that soft-loss variants produce extracted graphs closest in complexity to reference workflows across domains.

  • Problem

    Automatic extraction of workflows from dialog collections remains underexplored, although such workflows can support dialog-system design and domain-specific LLM grounding.

    Source: §1 · p. 1

  • Method

    Dialog2Flow embeds utterances by their communicative and informative actions, using soft contrastive learning informed by action-label semantics and twenty standardized TOD datasets.

    Source: §1 · p. 2, §3.1.2 · p. 3

  • Results

    D2F variants consistently outperform baselines across reported metrics, and D2Fsingle and D2Fjoint yield 6.86% and 8.57% average differences from reference graph complexity across domains, respectively.

    Source: §6.1 · p. 5, §7.2 · p. 9

  • Takeaways & Limitations

    Action-grouped embeddings can represent dialogs as trajectories whose clustering may uncover common conversational steps and underlying workflows.

    Source: §1 · p. 2, §7 · p. 7

    The study’s datasets are limited in size, and larger datasets are needed to further explore and validate the methods.

    Source: §9 · p. 10

Abstract

from arXiv · show

Efficiently deriving structured workflows from unannotated dialogs remains an underexplored and formidable challenge in computational linguistics. Automating this process could significantly accelerate the manual design of workflows in new domains and enable the grounding of large language models in domain-specific flowcharts, enhancing transparency and controllability. In this paper, we introduce Dialog2Flow (D2F) embeddings, which differ from conventional sentence embeddings by mapping utterances to a latent space where they are grouped according to their communicative and informative functions (i.e., the actions they represent). D2F allows for modeling dialogs as continuous trajectories in a latent space with distinct action-related regions. By clustering D2F embeddings, the latent space is quantized, and dialogs can be converted into sequences of region/action IDs, facilitating the extraction of the underlying workflow. To pre-train D2F, we build a comprehensive dataset by unifying twenty task-oriented dialog datasets with normalized per-turn action annotations. We also introduce a novel soft contrastive loss that leverages the semantic information of these actions to guide the representation learning process, showing superior performance compared to standard supervised contrastive loss. Evaluation against various sentence embeddings, including dialog-specific ones, demonstrates that D2F yields superior qualitative and quantitative results across diverse domains.

1 Introduction

The paper addresses automatic extraction of structured workflows from task-oriented dialogs, a comparatively underexplored problem with uses in dialog system design and LLM grounding. It proposes grouping utterances by their communicative and informative actions so clustered embeddings can reveal common conversational steps.

1 Introduction

  • Automatic workflow extraction from task-oriented dialogs is underexplored and could support dialog-system design, discourse analysis, data augmentation, human-agent training, and LLM grounding.The paper connects domain-specific workflow grounding with greater transparency and control.Source: §1 · p. 1
  • Dialog acts encode communicative intent and slots encode task-specific information; together, dialog actions define canonical steps that can reveal common workflows when aggregated or clustered.The proposed latent space groups utterances by representative actions rather than sentence semantics alone.Source: §1 · p. 2
  • The contributions are a standardized action-annotated collection of twenty TOD datasets, a semantic-action-based soft contrastive loss, and Dialog2Flow, a sentence embedding model pre-trained for dialog flow extraction.The authors report that the soft contrastive loss outperforms standard supervised contrastive loss.Source: §1 · p. 2

2 Related Work

Prior sentence embeddings commonly optimize semantic similarity, while dialog-specific work has found conversation-based similarity useful for TOD tasks. This paper motivates action-based similarity for workflow-related embeddings and builds on contrastive learning’s clustering objective.

Sentence Embeddings

  • Transformer sentence encoders pool token representations into a single embedding optimized for semantic similarity, but domains can require different notions of similarity.The cited dialog work reports conversation-based similarity outperforming semantic similarity across TOD tasks.Source: §2 · p. 2
  • TOD dialog models including TOD-BERT, DialogueCSE, and DSE motivate conversation-based similarity, while this work hypothesizes that action-based similarity can yield workflow-related sentence embeddings.Source: §2 · p. 2

Contrastive Learning

  • Contrastive learning clusters similar instances and separates dissimilar ones by minimizing anchor-positive distance and maximizing anchor-negative distance.Negatives are typically formed from other anchors’ positives within a mini-batch.Source: §2 · p. 2

3 Method

The method learns sentence representations with contrastive objectives, including a soft variant that uses semantic similarity between action labels to scale negative separation. It compares single combined action labels with separate dialog-act and slot targets.

3.1 Representation Learning Framework

  • A shared-weight BERT bi-encoder with mean pooling produces sentence representations, while a training-only MLP contrastive head maps them into the loss space and cosine similarity measures alignment.The normalized vectors place the latent space on a unit hypersphere.Source: §3.1 · p. 3

3.1.1 Supervised Contrastive Loss

  • Supervised contrastive loss groups same-label samples in a batch and uses temperature τ to control how strongly positive pairs are pulled together and negatives pushed apart.The mini-batch loss is averaged across its sample pairs.Source: §3.1.1 · p. 3

3.1.2 Supervised Soft Contrastive Loss

  • The soft contrastive loss uses semantic similarity between labels so anchors and negatives are separated proportionally to that similarity.Its temperature parameter controls the softness of negative labels.Source: §3.1.2 · p. 3

3.2 Training Targets

  • Training targets vary both the label decomposition and loss: D2F uses soft loss, while D2F-Hard uses supervised contrastive loss, with each using single act-plus-slots or joint act and slot targets.Joint targets use separate contrastive heads for dialog acts and slots.Source: §3.2 · p. 3

4 Training Corpus

The training corpus unifies twenty task-oriented dialog datasets after manual selection, annotation extraction, and standardization. Its per-turn annotations cover millions of utterances, standardized dialog acts, slots, and actions across dozens of domains.

4 Training Corpus

  • The authors identified twenty TOD datasets from which dialog-act and/or slot annotations could be extracted.Source: §4 · pp. 3–4
  • They manually inspected the datasets, extracted annotations, standardized domain names and dialog-act labels, and unified the data into a consistent format with per-turn annotations.Source: §4 · p. 4
  • The unified corpus contains 3.4 million utterances, 18 standardized dialog acts, 524 unique slot labels, and 3,982 action labels across 52 domains.Source: §4 · p. 4

5 Experimental Setup

The experiments use BERTbase for the main D2F results, compare against general and dialog-specific sentence-embedding baselines, and evaluate on unified task-oriented dialog data and SpokenWOZ.

5 Experimental Setup

  • D2F training follows the experimental setup of DSE and TOD-BERT, using BERTbase as the main-text encoder backbone.Additional configurations are reported in Appendix C, and implementation details are in Appendix B.Source: §5 · p. 4

5.1 Baselines

  • The baseline suite includes general sentence embeddings and dialog-specific models, including TOD-BERT, DSE, SBD-BERT, DialogGPT, and SPACE-2.Most dialog-context-optimized models may underperform on isolated sentences because they rely on dialog-specific features such as turns and roles.Source: §5.1 · p. 4, §5.2 · p. 4

5.2 Evaluation Data

  • Evaluation uses 26,910 Unified TOD utterances across 1,794 action labels and 31,303 SpokenWOZ utterances across 427 labels from complete single-domain conversations.SpokenWOZ conversations support later workflow extraction for seven domains; there are no single-domain calls for the profile domain.Source: §5.2 · p. 4

6 Similarity-based Evaluation

The evaluation tests whether embeddings organize utterances by dialog action using few-shot classification, anisotropy, and ranking. D2F variants outperform the baselines across reported metrics, while qualitative results show action-aware clustering and retrieval.

6 Similarity-based Evaluation

  • The evaluation measures local action classification with 1-shot and 5-shot prototypes, global geometry with intra- and inter-action anisotropy, and action ranking with nDCG@10.Classification reports macro F1 and accuracy; ranking retrieves the top-k closest embeddings for each action.Source: §6 · p. 5

6.1 Similarity-based Results

  • DSE outperforms general-purpose semantic embeddings on Table 2, while D2F's action-aware arrangement places related action clusters adjacent to one another.The examples include request price next to inform price, and inform name price between inform name and inform price.Source: §6.1 · pp. 5, 7, Figure 3 · p. 6
  • D2F has the strongest reported anisotropy gaps: 0.597 and 0.451 on the unified set, and 0.193 and 0.103 on SpokenWOZ, for single and joint targets.The authors interpret classification as a local view of representation quality and anisotropy as a more global view.Source: §6.1 · p. 5, Table 2 · p. 6, Table 3 · p. 6
  • D2F variants outperform baselines across classification and anisotropy metrics on both evaluation sets, and soft-contrastive D2F outperforms D2F-Hard.The reported tables cover the unified TOD evaluation set and SpokenWOZ; SpokenWOZ results show narrower differences and higher standard deviations than the unified set.Source: §6.1 · p. 5, Table 2 · p. 6, Table 3 · p. 6
  • D2F embeddings outperform baselines and D2F-Hard in ranking on both evaluation sets, despite increased variability and lower performance for all embedding types on SpokenWOZ.The ranking results use repeated samples of query utterances across actions.Source: Table 4 · p. 7, §6.1 · p. 7
  • In the SpokenWOZ phone-number example, D2F retrieves utterances expressing the same request, though slot-name differences make half count as incorrect.The authors note that clustering would still group the ten utterances because request phone_number and request phone are semantically equivalent actions.Source: §6.1 · p. 7

7 Dialog Flow Extraction Evaluation

The evaluation compares induced workflow graphs with reference graphs, measuring graph-size differences after clustering utterances without action annotations. D2F’s soft-contrastive variants produce the closest graph complexities across domains, while evaluation assumes the correct number of clusters is known.

7.1 Evaluation Details

  • Cross-domain slot-name mismatches partially affect SpokenWOZ results, and one cluster ID may correspond to multiple actions or vice versa, preventing direct graph comparison.The graph formulation also makes states individual actions and therefore non-Markovian, though the authors note that it remains easy to interpret.Source: §7.1 · pp. 7–8

7.2 Dialog Flow Extraction Results

  • 6.86% and 8.57% average difference for D2Fsingle and D2Fjoint, respectively, made their induced graphs closest in complexity to references across domains.Table 6 reports normalized absolute graph-size differences; lower values indicate a better match in graph complexity.Source: Table 6 · p. 8, §7.2 · p. 9

7.1 Evaluation Details

  • The induced graph is built without action annotations by clustering utterance embeddings and using cluster IDs as action labels; graph size is compared with the ground-truth graph by node-count difference.The reference is a weighted action-transition graph, and infrequent actions are pruned to reduce their influence on graph size.Source: §7 · p. 7, §7.1 · p. 8
  • For evaluation, each domain’s cluster count is set to its ground-truth action count, although the number of actions to cluster is unknown in practice.This gives all embedding models the same best-case setting and makes the input embeddings the only factor influencing the processed graph.Source: §7.1 · p. 8

7.2 Dialog Flow Extraction Results

  • 27.90% average difference across domains made DSE the strongest baseline, while other available embeddings tended to produce graphs with fewer actions than the references.The authors hypothesize that baseline embeddings group utterances by conversational context or semantic similarity, limiting the discovered dialog steps.Source: §7.2 · pp. 8–9

8 Conclusions

Dialog2Flow introduces embeddings trained to group task-oriented utterances by communicative and informative function, using a unified dataset of twenty datasets with standardized action annotations. The paper presents a hospital-domain induced graph as an example of a workflow that remains interpretable despite differing slightly from its reference.

8 Conclusions

  • D2F embeddings are pre-trained for dialog flow extraction, grouping utterances by their communicative and informative functions in a latent space.Training uses a unified dataset of twenty task-oriented dialog datasets with standardized action annotations, released with the work.Source: §8 · p. 9
  • Future work will explore larger backbone models and more sophisticated flow representations, including subtask graphs and dependency-parsing approaches.The proposed directions also include adapting dependency parsing to complex dialog structures.Source: §8 · p. 9
  • The authors provide source code to generate graphs for dialogue collections and embedding models, enabling manual assessment of D2F graph quality.They identify grounding LLMs in domain-specific flows and applying D2F to dialog state tracking and policy learning as potential applications.Source: §8 · p. 9
  • The D2Fjoint hospital graph has one fewer node than its reference yet still depicts the conversations’ common flow at a similar level of detail.The example traces greetings, a request for a department’s phone number, possible confirmation or clarification, and closing turns.Source: Figure 4 · p. 9

9 Limitations

The authors characterize the study as preliminary, focused on task-oriented dialogues and a relatively simple encoder. They identify limits in dialogue and domain coverage, dataset size, model complexity, and evaluation metrics.

9 Limitations

  • The study is restricted to task-oriented dialogues, so its methods and findings may not generalize well to more complex or diverse non-task-oriented conversations.The authors call for evaluation across a broader range of dialogue types.Source: §9 · p. 10
  • Training on a specific collection of domains, dialogue acts, and slots limits generalization to unseen domains and more varied interactions.The authors say broader training data is needed to improve robustness and applicability.Source: §9 · p. 10
  • The work uses a relatively standard encoder, and larger, more advanced models are a proposed avenue for improvement.Source: §9 · p. 10
  • The datasets, despite their scale, remain limited in size, leaving larger datasets necessary to further validate the methods.The authors also suggest named-entity tags as possible slots to broaden annotation beyond purely task-oriented dialogues.Source: §9 · p. 10
  • The employed metrics may miss aspects relevant to real-world performance, and dialog flow extraction lacks a standard metric.The authors call for better ways to represent and quantify flow quality.Source: §9 · p. 10

10 Ethical Considerations

The authors plan to release code and pre-trained weights under the MIT license, but not the unified dataset itself because of possible incompatibilities among source-dataset licenses. They also acknowledge possible gender bias encoded from the original data.

10 Ethical Considerations

  • Source code and pre-trained model weights are to be released under the MIT license for broad use and adaptation.Source: §10 · p. 10
  • The unified TOD dataset will not be released directly because the source datasets may have incompatible licenses; a script will let users select datasets to include.Source: §10 · p. 10
  • Gender bias in the original data may be partially encoded in the embeddings, including assumptions about an agent’s gender.The authors advise users to remain aware of this possibility and encourage further bias mitigation research.Source: §10 · p. 10

A Unified TOD Dataset

The authors unify annotated task-oriented dialog datasets into a standardized resource and describe its format and training configurations.

A Unified TOD Dataset

  • Twenty annotated TOD datasets were manually selected and unified, yielding 3.4 million utterances with 18 dialog acts, 524 slots, and 3,982 actions across 52 domains.Source: A Unified TOD Dataset · p. 15

A.1 Dataset Format

  • The common JSON format has a statistics header and a dialogs body containing utterances with speaker, text, domain, dialog-act, slot, and intent annotations.Source: §A.1 · p. 15

B Training Details

  • Training follows DSE and TOD-BERT settings, using BERTbase as the encoder backbone and a 128-dimensional contrastive head with 768-dimensional embeddings.Source: B Training Details · p. 15

Appendix C.

  • Appendix settings use semantic label similarity, τ′=0.35, learning rates 3e-4 and 3e-6, 15 epochs, maximum sequence length 64, and batch size 64.Source: Appendix C · p. 16

C Ablation study

The appendix tests D2F configurations, explains its soft contrastive objective, and examines temperature choice, cluster selection, and cluster labeling.

C Ablation study

  • Only replacing the backbone with pretrained DSE consistently improved F1 and anisotropy; adding self-supervision degraded performance, while removing the contrastive head caused a notable drop.Source: C Ablation study · p. 17

D Supervised Soft Contrastive Loss Explanation

E Soft Contrastive Loss Temperature

  • Soft contrast improves F1 and Δanisotropy over hard loss as τ′ rises, with a larger advantage for D2Fjoint, but the gains narrow with longer training.Metrics rise to around τ′=0.35–0.4; beyond that, anisotropy plateaus and F1 becomes less stable.Source: Figure A3 · p. 17, E Soft Contrastive Loss Temperature · p. 18

F How Many Actions to Cluster?

G Deriving Action Labels from Clusters

Related papers

Loading 2410.18481v2…