Source-linked AI summary
What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment
Hongyuan Mei, Mohit Bansal, Matthew R. Walter
TL;DR
Selective generation must choose salient records from over-determined databases while producing coherent natural-language descriptions. The paper introduces an end-to-end LSTM encoder-aligner-decoder with coarse-to-fine alignment, achieving state-of-the-art WEATHERGOV selection and generation and competitive or better ROBOCUP results without specialized resources.
Problem
Selective generation requires jointly deciding which records in an over-determined database to discuss and how to realize them as natural language.
Method
An end-to-end bidirectional LSTM encoder-aligner-decoder uses a coarse-to-fine aligner to select salient records and an LSTM decoder to generate descriptions directly from database-text pairs.
Results
The model achieves best-reported WEATHERGOV selection and generation results, including 12% relative F-1 and 59% relative BLEU improvements, and competitive or better ROBOCUP results.
Takeaways & Limitations
Joint end-to-end learning generalizes across domains without specialized features, linguistic resources, templates, or external selection labels.
Takeaways & Limitations
The model treats record sets as sequences in the dataset-specified order, and a different ordering may improve performance.
Abstract
from arXiv · showhide
We propose an end-to-end, domain-independent neural encoder-aligner-decoder model for selective generation, i.e., the joint task of content selection and surface realization. Our model first encodes a full set of over-determined database event records via an LSTM-based recurrent neural network, then utilizes a novel coarse-to-fine aligner to identify the small subset of salient records to talk about, and finally employs a decoder to generate free-form descriptions of the aligned, selected records. Our model achieves the best selection and generation results reported to-date (with 59% relative improvement in generation) on the benchmark WeatherGov dataset, despite using no specialized features or linguistic resources. Using an improved k-nearest neighbor beam filter helps further. We also perform a series of ablations and visualizations to elucidate the contributions of our key model components. Lastly, we evaluate the generalizability of our model on the RoboCup dataset, and get results that are competitive with or better than the state-of-the-art, despite being severely data-starved.
1 Introduction
Selective generation jointly chooses salient records from an over-determined database and realizes them as natural-language descriptions. The proposed end-to-end LSTM encoder-aligner-decoder uses coarse-to-fine alignment without specialized resources and achieves leading results on WEATHERGOV.
- Task: Selective generation combines content selection from an exhaustive database with surface realization of the chosen records.Joint learning is challenging because record relevance is ambiguous, selected records have complex dependencies, and descriptions vary.
- Approach: The model replaces separate, resource-intensive approaches with a neural encoder-aligner-decoder that jointly performs selection and free-form generation.It avoids specialized features, linguistic resources, semantic parsers, and generation templates.
- Architecture: A bidirectional LSTM encoder represents all event records, a coarse-to-fine aligner selects records through multiple abstractions, and an LSTM decoder generates their descriptions.The aligner uses pre-selection and refinement stages to avoid searching the full over-determined record set.
- Alignment: The alignment mechanism learns to associate each generated word with an event record, supporting joint content selection and surface realization.The end-to-end framework trains directly on paired record sets and descriptions without ground-truth content-selection labels.
- Results: 12% relative F-1 improvement in selection and 59% relative BLEU improvement in generation are reported on WEATHERGOV without domain-specific resources.The model achieves the best results reported to date on both tasks.
2 Related Work
Earlier selective-generation research often separated content selection from realization or relied on domain-specific resources and templates. This work instead frames the full task as end-to-end neural learning with recurrent alignment and coarse-to-fine search.
- Prior Work: Selective generation has historically received less attention as a unified problem than its individual content-selection and realization subproblems.Prior work includes content-structure modeling, collective classification, and record-to-text alignment.
- Surface Realization: Surface-realization systems commonly generate text from formal meaning representations using grammars or probabilistic string formalisms.Examples include WIDL representations, synchronous context-free grammars, and probabilistic context-free grammars.
- Unified Systems: Unified selective-generation systems have used translation models, staged frameworks, or log-linear sequence decisions, often with external alignments or domain-specific features.These approaches include Chen and Mooney, Kim and Mooney, and Angeli et al.
- Positioning: Konstas and Lapata jointly optimize selection and realization with a probabilistic grammar, but record selection and ordering remain locally Markovized.The present approach is positioned as an alternative to this local chaining strategy.
- This Work: The proposed recurrent encoder-aligner-decoder learns both tasks directly from database-text pairs without external aligners, selection labels, manual features, templates, or parsers.Its coarse-to-fine aligner divides search into pre-selection and refinement stages to handle over-determined databases.
3 Task Definition
The task maps an over-determined database representation of a world state to a natural-language description. Training uses paired complete record sets and texts, while evaluation covers WEATHERGOV and data-scarce ROBOCUP scenarios.
- Task Definition: The input is a rich world state represented by an over-determined set of database event records, and the output is a natural-language description.The task requires deciding which records to discuss and how to describe them.
- Data Format: Training data consists of scenario pairs (r(i), x(i)), where r(i) is the complete record set and x(i) is its natural-language description.At test time, only the records are provided.
- Examples: Figure 1 presents sample database-text pairs from WEATHERGOV and ROBOCUP.The examples illustrate the two benchmark domains used for evaluation.
- WEATHERGOV: WEATHERGOV contains 29,528 scenarios with 36 weather records per scenario, paired with forecasts averaging 28.7 words.Records include attributes such as temperature and sky cover.
- ROBOCUP: ROBOCUP contains 1,539 pairs of temporally ordered robot-soccer events and commentary, averaging 2.4 records and 5.7 words per scenario.The dataset is used to assess generalizability in a data-starved domain.
4 The Model
The model jointly selects salient records and generates descriptions through a bidirectional LSTM encoder, coarse-to-fine aligner, and LSTM decoder. Its aligner uses multiple input abstractions and two-stage reweighting to make selective generation differentiable and less distracted by non-salient records.
- Encoder: The encoder represents the input records with bidirectional LSTM hidden annotations that model dependencies among records.Each annotation summarizes one record while incorporating forward and backward context.
- Coarse-to-Fine Aligner: The coarse-to-fine aligner performs content selection independently of input ordering by reasoning over both records and their hidden annotations.This multi-level representation supports selection from an over-determined record set.
- Coarse-to-Fine Aligner: The pre-selector assigns each record a selection probability, and the refiner reweights decoding-time alignment weights using those probabilities.The two stages are a pre-selector followed by a refiner, avoiding search over the full record set.
- Coarse-to-Fine Aligner: The pre-selector prior discourages attention from selecting non-salient records and can approximate the number of selected records through the sum of its probabilities.The target count γ regularizes this sum during training.
- Decoder and Training: The LSTM decoder conditions next-word probabilities on the current context vector, previous word, and previous decoder state.Training maximizes ground-truth description likelihood with regularization for selection-count control and at least one pre-selected record.
- Inference: Inference generates descriptions by maximum a posteriori decoding; greedy search was used because beam search did not improve performance on the considered datasets.The model treats records as a sequence using the dataset-specified order, although alternative orderings may improve performance.
5 Experimental Setup
The experiments evaluate selection and surface realization on WEATHERGOV and test generalizability on data-starved ROBOCUP. Selection uses F-1, while generation uses BLEU and dataset-specific BLEU variants.
- Datasets: The study evaluates WEATHERGOV and uses the data-starved ROBOCUP dataset to assess generalizability.WEATHERGOV uses 25,000 training, 1,000 development, and 3,528 test scenarios; ROBOCUP uses three-fold cross-validation across games.
- Training Details: WEATHERGOV tuning selects 500 hidden units and γ = 8.5 using development-set BLEU, while ROBOCUP tunes only γ = 5.0.The hidden-unit count is not retuned on ROBOCUP.
- Evaluation Metrics: Content selection is evaluated with F-1 over selected records, defined as the harmonic mean of precision and recall against ground-truth selections.The selected set contains the record with the largest aligner selection weight at each decoding step.
- Evaluation Metrics: Surface realization is evaluated with BLEU against human references and, on WEATHERGOV, cBLEU that tolerates numerical deviations of at most five.On ROBOCUP, the study also evaluates an additional BLEU-based metric, though the supplied passage truncates its name.
6 Results and Analysis
The model achieves state-of-the-art WEATHERGOV selection and generation results, while ablations, qualitative analyses, and ROBOCUP experiments examine its components, behavior, and domain generalizability.
- Primary Results (WEATHERGOV): 11.94% relative F-1, 58.88% relative sBLEU, and 36.68% relative cBLEU improvements over prior state-of-the-art results were achieved on WEATHERGOV.The evaluation covers content selection and surface realization using F-1, standard sBLEU, and customized cBLEU.
- Beam Filter with k-Nearest Neighbors: The k-nearest-neighbor beam filter improves results over the primary greedy-search results, whereas standard beam search performs worse.The filter is evaluated as an alternative decoding approach after beam search underperforms on the dataset.
- Ablation Analysis (WEATHERGOV): The coarse-to-fine aligner yields superior F-1 and BLEU scores relative to a standard aligner in the WEATHERGOV ablation.The comparison is reported on the development set.
- Ablation Analysis (WEATHERGOV): Encoding event records with an LSTM-RNN produces a significant gain compared with omitting the encoder, attributed to capturing relationships among records.This encoder ablation is evaluated on the development set.
- Qualitative Analysis (WEATHERGOV): The qualitative example achieves standard BLEU 100.00 and shows record-word alignments that identify salient records and associate fields with generated words.Examples include windDir with “southeast,” temperature with “71,” windSpeed with “wind 10,” and gust with the corresponding gust phrase.
- Out-of-Domain Results (ROBOCUP): 25.28 sBLEU and 81.58 selection F-1 on ROBOCUP exceed the previous best or are the best reported results, while 29.40 sBLEUG is competitive with the state-of-the-art.The dataset has approximately 1000 training pairs, and the experiment uses an ensemble of five randomly initialized models.
7 Conclusion
The paper presents a specialized-feature-free encoder-aligner-decoder model for selective generation, achieving strong benchmark performance and generalizing to a data-starved domain.
- The model uses no specialized features, linguistic resources, or generation templates.
- The model jointly learns content selection and surface realization with a bidirectional LSTM-RNN and novel coarse-to-fine aligner.
- It achieves state-of-the-art selection and generation results on WEATHERGOV and further improves through a k-nearest neighbor beam filter.
- On ROBOCUP, the model achieves results competitive with or better than the state-of-the-art despite the data-starved setting.
A Supplementary Material
The supplement provides further evaluations of the model beyond the original manuscript.
- The supplementary material contains additional evaluations of the model.
- These evaluations are presented as a supplement to the original manuscript.
- The passage identifies the material as evaluation-focused rather than introducing a new model component.
A.1 Beam Filter with k-Nearest Neighbors
The supplementary beam-search analysis compares greedy inference with a k-nearest-neighbor beam filter, which improves BLEU over the primary greedy results.
- Greedy search approximates full inference and yields better BLEU performance than beam search on this dataset.
- The k-nearest-neighbor beam filter selects among M-best beam candidates using BLEU against descriptions from K nearest training neighbors.
- With tuned settings M = 2 and K = 1, the test-time filter achieves BLEU scores better than the primary greedy results.
A.2 Word Embeddings (Trained & Pretrained)
The supplementary embedding analysis examines semantic structure in learned decoder embeddings and compares two ways to incorporate pretrained word vectors.
- The decoder learns word embeddings through the embedding matrix E during training.
- Nearest-neighbor words in cosine similarity space are used to inspect semantic relationships among WEATHERGOV training words.
- Pretrained embeddings are either used to initialize and refine the learned matrix or concatenated with it.