Source-linked AI summary
Learning Neural Templates for Text Generation
Sam Wiseman, Stuart M. Shieber, Alexander M. Rush
TL;DR
Encoder-decoder NLG models are effective but largely uninterpretable and difficult to control in phrasing or content. The paper introduces a neural HSMM decoder that jointly learns discrete template-like structures and generation, achieving competitive performance while making generations more interpretable and controllable.
Problem
Encoder-decoder NLG models sacrifice interpretable outputs and controllable form or content despite improving fluency and reducing manual effort.
Method
A neural HSMM decoder jointly learns discrete, template-like structures and generates text conditioned on the source.
Results
The model achieves performance competitive with other neural NLG approaches while inducing templates that support interpretation and control.
Takeaways & Limitations
Learned templates provide explicit representations of intended content and expression, and can be manipulated to control and interpret generations.
Takeaways & Limitations
The model assumes segments are independent given their latent variable and source x, although the encoder may capture needed interdependence.
Abstract
from arXiv · showhide
While neural, encoder-decoder models have had significant empirical success in text generation, there remain several unaddressed problems with this style of generation. Encoder-decoder models are largely (a) uninterpretable, and (b) difficult to control in terms of their phrasing or content. This work proposes a neural generation system using a hidden semi-markov model (HSMM) decoder, which learns latent, discrete templates jointly with learning to generate. We show that this model learns useful templates, and that these templates make generation both more interpretable and controllable. Furthermore, we show that this approach scales to real data sets and achieves strong performance nearing that of encoder-decoder text generation models.
1 Introduction
The paper proposes a neural generation model that learns discrete, template-like structures with an HSMM decoder to improve interpretability and controllability while retaining competitive performance.
- Encoder-decoder NLG models improve fluency and reduce manual effort but sacrifice interpretable outputs and controllable form or content.
- The proposed system learns discrete, template-like structures for conditional text generation with a neural HSMM decoder.Training is performed in a data-driven manner by backpropagation through inference.
- Learned templates explicitly represent intended content and the way that content is expressed through instantiated templates.
- The model achieves competitive performance with neural NLG approaches while enabling template-based interpretation and control.Experiments indicate that templates can be induced and manipulated to control and interpret generations.
2 Related Work
Prior NLG work includes manually engineered or automatically induced templates, probabilistic template models, black-box neural generation, and approaches to controllable text generation.
- Traditional and automated NLG systems use hand-engineered templates or induce templates by clustering sentences and abstracting fields.
- Probabilistic template approaches have been paired with discriminative classifiers or rerankers and explicitly model knowledge-base field selection.The paper instead presents an end-to-end model over generation segments.
- Most neural NLG work uses unconstrained black-box encoder-decoder models, alongside research on discrete variables and non-HSMM segmental models.
- Controllable text generation commonly manipulates attributes such as sentiment, whereas this work controls content or expression by manipulating latent templates.
3 Overview: Data-Driven NLG
The paper frames data-to-text NLG as generating fluent descriptions from knowledge-base records, contrasting opaque encoder-decoder generation with explicit template-like planning.
- The task is to generate an adequate and fluent textual description from a collection of typed knowledge-base records.The records contain a type, entity, and value, and the experiments use E2E and WikiBio datasets.
- Neural NLG commonly encodes the knowledge base and generates the description word by word through a black-box conditional decoder.
- A WikiBio example pairs a database for Frederick Parker-Rhodes with a reference generation, illustrating the database-to-text setting.
- Because decoder states obscure which source aspects relate to outputs, neural generation is difficult to control finely and to interpret when mistakes occur.
- Filtering source information in a standard encoder-decoder can produce unexpected output changes that propagate through the system.
- Traditional systems make intent explicit through planning and surface realization, while the proposed model uses induced templates as fixed content plans linked to records.
4 Background: Semi-Markov Models
An HSMM models output sequences through latent segments whose states and lengths determine transitions and emissions, providing a probabilistic basis for template-like generation.
- A template can be represented as a sequence of typed text segments, with some segments forming a backbone and others filled from the knowledge base.
- An HSMM extends an HMM by allowing emissions to span multiple time steps and to have dependencies within a segment.
- The model assigns each output timestep a discrete latent state, a segment-length variable, and a binary segment-finish indicator.
- A conditional HSMM conditions segment generation on the source x, functioning as an HSMM decoder.
- The HSMM likelihood combines state transitions, segment lengths, and observations generated for each segment.The transition distribution factors into a next-state probability conditioned on the current state and source and a length probability conditioned on the next state.
5 A Neural HSMM Decoder
The neural HSMM decoder combines neural sequence-generation components with latent segment states and lengths, enabling learned template-like structures. Dynamic-program inference supports training and template-conditioned generation, making outputs more interpretable and controllable while exposing a segment-independence limitation.
- 5.1 Parameterization: The decoder parameterizes an HSMM with neural components, including LSTMs and attention, while preserving latent segmentation structure.The transition model depends on latent states and encoded source information; emissions use state-conditioned RNNs with attention and copying.
- 5.1 Parameterization: Transition probabilities form a source-conditioned K × K state matrix, while segment lengths are fixed to a uniform distribution up to maximum length L.The model uses nonlinear functions of the source representation for transitions; uniform lengths encourage functionally similar emissions of different lengths to cluster together.
- 5.1 Parameterization: Segment emissions are products of token probabilities from a latent-state-conditioned RNN, with attention and copy-attention over input records.An autoregressive variant additionally conditions each next-token distribution on all previously generated tokens, but not on tokens from other segments.
- 5.2 Learning: The model marginalizes unobserved states, lengths, and boundaries using an HSMM dynamic program and trains by backpropagating through that differentiable inference procedure.The backward algorithm computes the marginal likelihood, and automatic differentiation is used to optimize the neural parameters.
- 5.4 Discussion: The structured decoder supports more interpretable and controllable generations, but assumes segments are independent given their latent variables and source x.The source encoder may still capture inter-segment dependencies, such as the correct determiner for an upcoming noun phrase.
- 5.3 Extracting Templates and Generating: Viterbi segmentations associate text phrases with discrete states, yielding templates such as one with S = 17 segments that can guide constrained beam-search generation.A chosen extracted template can be explicitly imposed, and each generated segment is typed by its corresponding latent variable.
6 Data and Methods
Experiments evaluate the neural HSMM on E2E and WikiBio, using preprocessing, constrained learning, tied emissions, and autoregressive variants alongside published and template baselines.
- Datasets and evaluation: The approach is evaluated on the E2E and WikiBio data-to-text datasets against published encoder-decoder and template-style baselines.E2E uses BLEU, NIST, ROUGE, CIDEr, and METEOR; WikiBio uses BLEU, NIST, and ROUGE.
- Preprocessing: Multiword record values are split into positional single-word records because the emission model uses a word-level copy mechanism.For example, type[coffee shop] becomes type-1[coffee] and type-2[shop].
- Training: Constraining segmentations to respect word spans appearing in records yields hard-constrained MAP inference through the model’s objective.
- Architecture: Emission distributions are tied across multiple states to reduce the cost of computing K vocabulary distributions while retaining distinct transition distributions.
- Model variants: The experiments include non-autoregressive and autoregressive HSMMs, with state counts selected using BLEU on held-out validation data.The best configurations use K = 275 or 60 for E2E and K = 135 for WikiBio.
7 Results
The neural HSMM is competitive with neural baselines on automatic metrics and supports interpretable, controllable generation through learned discrete templates and states.
- Automatic evaluation: The proposed model is fairly competitive with neural models, while templated baselines generally underperform neural systems.
- Automatic evaluation: On E2E validation data, neural HSMM models are largely competitive with the encoder-decoder system, although the gap increases on test data.
- Automatic evaluation: On WikiBio, HSMMs are competitive with the best Lebret et al. model and outperform it on ROUGE, while later seq2seq results surpass both.
- Controllable diversity: Varying the template while holding the database fixed changes word ordering and which fields are mentioned in generated text.
- Interpretable states: Discrete states type generated segments and consistently correspond to particular pieces of information, supporting interpretation of the generation.
- Interpretable states: Average purity measures quantify how consistently learned discrete states align with record types on E2E and WikiBio.
8 Conclusion and Future Work
The paper presents a scalable neural HSMM template model that learns discrete latent structures for controllable and interpretable text generation, while identifying broader extensions for future work.
- Conclusion: The neural HSMM learns template-like latent state sequences tractably by backpropagating through a dynamic program.
- Conclusion: The approach scales to large-scale text datasets and is nearly competitive with encoder-decoder models.
- Conclusion: Learned templates support controllable generation diversity and interpretable states during generation.
- Future work: Future work includes encouraging more different or minimal templates, modeling paragraphs and documents, and developing hierarchical templates.
A.1 Additional Model and Training Details
The model embeds records and source information, uses LSTM-based components for decoding and copying, and trains the HSMM with SGD and dropout.
- Record representations combine type, position, and word-value features through an MLP with ReLU.
- Copied tokens condition the decoder LSTM on record features, while ambiguous copy sources are averaged during training and resolved with MAP selection at test time.
- The autoregressive HSMM adds a one-layer LSTM, with token-history inputs working slightly better on WikiBio and copied-field-type averages working slightly better on E2E.
- The transition function C(xu) uses nested ReLU transformations to produce source-conditional hidden-state embeddings.
- The transition parameterization uses dimensions m1 = 64, m2 = 32, and m3 = 64 in all experiments.
- Training uses SGD with learning rate 0.5, epoch-based decay after stalled validation log-likelihood, delayed optimization of the autoregressive LSTM, and dropout regularization.
A.2 Additional Learned Templates
The appendix visualizes additional templates learned on E2E and WikiBio by both non-autoregressive and autoregressive HSMM models.
- Selection procedure: Five dissimilar templates are selected from the 200 most frequent templates using greedy coverage of previously unseen states.
- E2E templates: Table 6 presents five templates extracted from E2E data for the NTemp and Ntemp+AR models.
- WikiBio templates: Table 7 presents five templates extracted from WikiBio data for the NTemp and Ntemp+AR models.