Source-linked AI summary
Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention
Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, William Yang Wang
TL;DR
Existing semantically controlled response models face a scalability challenge as dialog-act combinations grow across domains. The paper represents dialog acts as paths in a hierarchical graph and encodes that structure in HDSA; on MultiWOZ, it reports stronger performance than competing algorithms, alongside improved generalization and sample complexity outcomes.
Problem
Method
The model represents dialog acts as paths through a hierarchical graph and uses hierarchical disentangled self-attention heads to model designated graph nodes and semantic conditions.
Source: §1 · p. 1, §3 · p. 4, §4 · pp. 5–6Results
On large-scale MultiWOZ, HDSA significantly outperforms competing algorithms and improves unseen-case generalization while decreasing sample complexity on seen cases.
Source: §1 · p. 2Takeaways & Limitations
The authors report that structured dialog-act modeling improves generalization on unseen cases and decreases sample complexity on seen cases.
Source: §1 · p. 2The method is studied only in the supervised setting with annotated dialog acts; response generation without those annotations is not investigated.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Semantically controlled neural response generation on limited-domain has achieved great performance. However, moving towards multi-domain large-scale scenarios are shown to be difficult because the possible combinations of semantic inputs grow exponentially with the number of domains. To alleviate such scalability issue, we exploit the structure of dialog acts to build a multi-layer hierarchical graph, where each act is represented as a root-to-leaf route on the graph. Then, we incorporate such graph structure prior as an inductive bias to build a hierarchical disentangled self-attention network, where we disentangle attention heads to model designated nodes on the dialog act graph. By activating different (disentangled) heads at each layer, combinatorially many dialog act semantics can be modeled to control the neural response generation. On the large-scale Multi-Domain-WOZ dataset, our model can yield a significant improvement over the baselines on various automatic and human evaluation metrics.
1 Introduction
The introduction frames multi-domain response generation as difficult because dialog-act combinations grow combinatorially. It proposes a graph-structured act representation and HDSA, reporting better baseline performance, generalization, and sample complexity results on MultiWOZ.
1 Introduction
- Semantically controlled models perform well on limited domains, but possible dialog acts grow combinatorially as the number of domains increases.Traditional slot-filling systems also require substantial human handcrafting and can struggle to generate natural, generalizable utterances.Source: §1 · p. 1
- The proposed graph merges nodes with the same semantic meaning so each dialog act is a root-to-leaf route, reducing representation cost to almost linear.Figure 2 depicts graph paths as dialog acts and illustrates the corresponding HDSA head activations.Source: §1 · p. 1, Figure 2 · p. 2
- HDSA binds attention heads to graph nodes and activates heads matching the dialog act path during training and generation.At test time, predicted acts activate the heads to control response semantics without handcrafting rules.Source: §1 · pp. 1–2
- HDSA significantly outperforms competing algorithms on large-scale MultiWOZ, with improved unseen-case generalization and lower sample complexity on seen cases.The introduction reports these outcomes without supplying metric values.Source: §1 · p. 2
2 Related Work & Background
The background describes task-oriented dialogue systems as modular pipelines and identifies the large semantic space of MultiWOZ as a challenge. This paper focuses on act prediction and response generation, using HDSA to model semantic conditions with separate attention heads.
2 Related Work & Background
- Canonical task-oriented dialog systems use separate modules for intention classification, belief tracking, dialog-act prediction, and response generation.Responses may be delexicalized with database placeholders and then post-processed with database records.Source: §2 · p. 2, Figure 3 · p. 3
- MultiWOZ's many slots and complex ontology challenge existing approaches, leaving its large semantic space a difficult research problem.The paper focuses on act prediction and response generation, assuming ground-truth belief states and database records.Source: §2 · p. 3
- Unlike work combining self-attention with multiple NLP tasks, HDSA disentangles heads to model different semantic conditions within a single task.The authors associate this design with greater controllability and interpretability.Source: §2 · p. 3
3 Dialog Act Representation
Dialog acts encode semantics at several granularities, and this section represents each act through a compact graph that shares semantically identical cross-branch nodes.
3 Dialog Act Representation
- Dialog acts form hierarchical paths, but concatenating one-hot encodings across tree levels creates flat representations that impede transfer and can raise sample complexity.The canonical flat encoding is used in SC-LSTM (Wen et al., 2015; Budzianowski et al., 2018).Source: §3 · p. 3
- Merging cross-branch nodes with the same semantics yields a compact acyclic graph representation of dialog acts.Each act is a root-to-leaf route; a dummy “none” node converts non-leaf acts into triplets.Source: §3 · pp. 3–4
- Multiple dialog-act graph representations for one response are aggregated with a bit-wise OR into a single representation.The paper denotes this aggregate as A.Source: §3 · p. 4
- Under strong cross-branch overlap, the graph lowers sample complexity, and separately observed components may support generalization to the unseen “hotel-recommend-area” act.The example’s component semantics may have appeared separately in other acts.Source: §3 · p. 4
4 Model
The model predicts hierarchical dialog acts from dialog history and uses their graph representation to switch semantic attention heads during response generation. Stacked disentangled self-attention layers build responses from coarse- to fine-grained semantics.
Dialog Act Predictor
- The act predictor encodes dialog history into token-wise and overall representations, then predicts activation probabilities for dialog-act graph nodes and thresholds them into predicted acts.The overall representation is used to predict the hierarchical act representation.Source: §4 · pp. 4–5
Disentangled Self-Attention
- DSA computes independent self-attention heads and uses a binary switch to sum selected head outputs instead of concatenating them as in a standard Transformer.The heads share subsequent cross-attention, feed-forward, normalization, and projection layers; the switch controls information flow.Source: §4 · pp. 5–6
Hierarchical DSA
- HDSA stacks DSA layers so lower layers capture coarse semantics and upper layers capture fine-level semantics.The final layer output is used to compute the response sequence probabilities.Source: §4 · p. 6
Dialog Act Predictor
- The hierarchical graph is encoded as a prior: each HDSA head models a designated semantic node, and graph representation A controls layer switches.The predicted acts control generation at test time; the illustrated model uses three layers with 4, 3, and 6 heads.Source: Figure 5 · p. 5, §4 · p. 6
5 Experiments
Experiments on MultiWOZ evaluate dialog-act prediction, response generation, generalization across act frequencies, response quality, and semantic controllability. The results favor hierarchical graph-based representations and HDSA over the tested alternatives.
5 Experiments
- MultiWOZ supplies 625 potential dialog acts represented by a three-layer graph with 44 nodes; the study selects 1,000 dialogs each for development and test.The experiments focus mainly on context-to-response generation, with dialog-act prediction treated as a preliminary task.Source: §5 · pp. 6–7, Table 1 · p. 7
- Fine-tuned BERT performs significantly better than the other tested dialog-act predictors and is fixed during generator training.The predictor and response generator are not jointly trained.Source: §5 · p. 7, Table 2 · p. 7
5.1 Automatic Evaluation
- 10+% Inform/Request success-rate gains are reported for the proposed approaches; ground-truth acts add 7.0 BLEU and 21% Entity F1 for HDSA.The reported ground-truth-act gains are relative to the predicted-act setting.Source: Table 3 · p. 8, §5.1 · p. 8
- The graph representation improves results across different models, and structure-aware HDSA gains over Transformer-in/out with flattened graph input.The authors also report restricted gains from sparse tree acts and an additional gain when HDSA receives ground-truth acts.Source: §5.1 · p. 8
- For few-shot dialog acts, Transformer-in with compact Graph-Act significantly boosts BLEU over Transformer-in with sparse Tree-Act; HDSA also achieves better results than Graph-Act Transformer-in by exploiting hierarchical dialog-act structure.Source: §5.1 · p. 8
5.2 Human Evaluation
- Human judges rate HDSA significantly above SC-LSTM on coherence with dialog history.The comparison assessed relevance, coherence, and consistency using responses generated by both models and the reference history.Source: Table 4 · p. 9, §5.2 · p. 9
- Human controllability results indicate that both compact dialog-act representation and the hierarchical structure prior matter for following semantic conditions.The synthetic evaluation used randomly selected dialog histories and combinations of dialog acts as conditions.Source: Table 4 · p. 9, §5.2 · p. 9
6 Discussion
The graph representation supports transfer across overlapping domain-slot concepts, but compression can make some act combinations ambiguous.
Graph Representation as Transfer Learning
- Grouping similar concepts under a hypernym with one switch is proposed for domains with little exact domain-slot overlap.The paper notes that transfer is easier when domain-slot-value pairs overlap substantially, as in Restaurant and Hotel.Source: Graph Representation as Transfer… · p. 9
- Overlapping actions can merge distinct acts into an ambiguous route; the authors say these unnatural cases are rare in the dataset but leave expressiveness unresolved.Their example merges restaurant-inform-price and hotel-inform-location, making them indistinguishable from hotel-inform-price or restaurant-inform-location.Source: Graph Representation as Transfer… · p. 9
7 Conclusion and Future Work
The conclusion frames the framework as addressing scalability and generalization, while identifying annotated dialog acts as a current setting constraint.
7 Conclusion and Future Work
- The method is evaluated only in a supervised setting with annotated dialog acts; inferring acts from annotated responses is future work.The authors propose using such inferred, potentially noisy data to guide response generation.Source: §7 · p. 9
A Details of Model Implementation
The implementation uses a three-layer Transformer encoder and a three-layer self-attention decoder with layer-specific head dimensions.
A Details of Model Implementation
- The encoder uses three Transformer layers with 64-dimensional embeddings and four heads, then uses the first symbol as sentence representation u to score tree nodes for dialog-act prediction.Each head has query, key, and value dimensions of 16.Source: A Details of Model Implementation · p. 12, Figure 8 · p. 12
- The decoder takes 64-dimensional input features and uses 10 heads with per-head dimensions 6, 9, and 2 across its three layers.Values have dimension 16; the feed-forward layer projects features to 64 dimensions and then to the 3.1K vocabulary.Source: A Details of Model Implementation · p. 12
B Automatic Evaluation
The section demonstrates an example of the paper’s automatic evaluation metrics in Figure 9.
B Automatic Evaluation
- Figure 9 demonstrates an example of the automatic evaluation metrics.Source: B Automatic Evaluation · p. 12
C Baseline Implementation
The baseline feeds dialog-act input as an embedding into the transformer to control sequence generation; the human-evaluation interface is illustrated separately.
C Baseline Implementation
- Dialog-act input is fed as an embedding into the transformer to control sequence generation.Source: C Baseline Implementation · p. 12
D Human Evaluation Interface
- Figure 10 illustrates the user interface used in the human evaluation procedure.Source: D Human Evaluation Interface · p. 12
E Controllability Evaluation
The controllability example illustrates responses constrained by dialog-act conditions, while the appendix enumerates the graph’s semantic choices and provides visual guides.
E Controllability Evaluation
- The example uses three dialog acts as semantic conditions for constrained response generation; Figure 11 marks passes and failures with checks and crosses.Source: E Controllability Evaluation · p. 12, Figure 11 · p. 13
F Enumeration of all the Dialog Acts
- The graph enumerates 10 domain choices, 7 action choices, and 27 slot choices.Source: F Enumeration of all the Dialog Acts · p. 12
E Controllability Evaluation
- Figure 9 illustrates evaluation metrics in both delexicalized and non-delexicalized forms.Source: Figure 9 · p. 13
- Figure 10 illustrates the human-evaluation interface.Source: Figure 10 · p. 13
F Enumeration of all the Dialog Acts
- Figure 12 illustrates the entire dialog graph, which the text then enumerates.Source: F Enumeration of all the Dialog Acts · p. 12, Figure 12 · p. 14