Source-linked AI summary
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Amit Moryossef, Yoav Goldberg, Ido Dagan
TL;DR
Data-to-text systems often conflate planning with realization, creating faithfulness and coherence challenges. The paper separates them with symbolic planning and neural realization, finding comparable fluency but substantially better faithfulness and controllable, diverse outputs.
Problem
Neural data-to-text systems combine planning and realization, yet can struggle with coherent ordering and faithfulness to input facts.
Method
The approach generates explicit symbolic text plans, matches reference texts to plans for training, and ranks possible plans at inference before neural realization.
Results
The plan-based system is on par with a strong end-to-end neural system on automatic metrics and fluency, while substantially outperforming it on faithfulness.
Takeaways & Limitations
Separating planning from realization supports explicit control of output structure and generation of diverse sentences without sacrificing fluency.
Takeaways & Limitations
Plan construction is dataset dependent and may not generalize as is to other datasets or more demanding generation tasks.
Abstract
from arXiv · showhide
Data-to-text generation can be conceptually divided into two parts: ordering and structuring the information (planning), and generating fluent language describing the information (realization). Modern neural generation systems conflate these two steps into a single end-to-end differentiable system. We propose to split the generation process into a symbolic text-planning stage that is faithful to the input, followed by a neural generation stage that focuses only on realization. For training a plan-to-text generator, we present a method for matching reference texts to their corresponding text plans. For inference time, we describe a method for selecting high-quality text plans for new inputs. We implement and evaluate our approach on the WebNLG benchmark. Our results demonstrate that decoupling text planning from neural realization indeed improves the system's reliability and adequacy while maintaining fluent output. We observe improvements both in BLEU scores and in manual evaluations. Another benefit of our approach is the ability to output diverse realizations of the same input, paving the way to explicit control over the generated text structure.
1 Introduction
Data-to-text generation varies in both information structure and verbalization, but neural systems commonly combine planning and realization, risking inadequate or incoherent outputs. The paper proposes an explicit symbolic planner followed by neural realization to improve faithfulness while preserving fluency.
- Text planning chooses fact order, entity order, and sentence splits, while plan realization verbalizes a fixed structure.
- For two facts, fact ordering, entity ordering, and sentence splits produce 12 possible structures, with more facts creating many more possibilities.
- Recent neural NLG systems treat generation as one end-to-end mapping from input facts to output text, rather than explicitly separating planning from realization.
- Neural systems are fluent but can lose coherence, produce incoherent fact orders, and omit, repeat, hallucinate, or change input facts.
- The proposed symbolic planner generates ordered trees guaranteed faithful and complete with respect to input facts, which a neural system then realizes as fluent text.
- On WebNLG, the approach produces outputs as fluent as neural systems but more faithful to the input facts, while enabling structural control and diverse outputs.
2 Overview of the Approach
The approach represents RDF-set inputs as graphs and separates non-neural planning from neural sentence realization. It constructs matching plans for training and acknowledges that the concrete plan-construction method is dataset dependent.
- Each input is an RDF set represented as a graph of entities and directed labeled relations, paired with one or more reference texts.
- The non-neural data-to-plan step specifies sentence divisions, fact order within sentences, and sentence order before NMT realizes each sentence.
- The method annotates input-reference pairs with corresponding plans and uses the resulting dataset to train plan-to-text translation and select test-time plans.
- Figure 1 traces the pipeline from RDF triplets to an explicit plan, linearization, and neural generation of the output text.
- The plan-construction method is dataset dependent and may not generalize unchanged to other datasets or more demanding generation tasks.
3 Text Planning
Text planning represents how input facts are grouped, ordered, directed, and structurally connected before realization. The paper recovers plans from references and ranks candidate plans using learned scoring factors.
- Plan structure: A text plan divides facts into sentences and orders both sentences and facts within each sentence.
- Plan structure: Plans also specify relation direction and tree structures connecting facts through shared entities, including chains and siblings.Chains share a middle entity, while siblings share a parent.
- Plan structure: Each input fact must be expressed exactly once in a matching plan, preserving faithfulness and completeness with respect to the input graph.
- Adding Plans to Training Data: Reference-plan matching uses sentence splits and recognized entity order, while relaxing exact matching for unrecognized or ambiguous textual mentions.Entity matching is difficult because names may vary, require external knowledge, or involve coreference.
- Adding Plans to Training Data: The method skips cyclic graphs and assumes each entity is mentioned only once per sentence, conditions tied to the WebNLG setting.
- Test-time Plan Selection: Candidate plans are ranked with a product-of-experts model using relation direction, relation transitions, and other learned plan properties.Transition probabilities capture tendencies for related relations to appear together and support aggregation.
- Test-time Plan Selection: Highly ranked plans generally produce adequate texts, and realizing multiple high-scoring plans enables diverse outputs for the same input.
4 Plan Realization
The realization stage translates symbolic sentence plans into fluent text with a neural machine translation system. Sentence-level realization and entity copying help the model follow the prescribed plan.
- Plan realization: An off-the-shelf NMT system realizes each sentence plan individually, reducing the information that must be remembered during generation.The authors expect this division to reduce over- and under-generation of facts.
- Plan realization: Sentence plans are linearized through pre-order tree traversal with brackets marking structure and tokens encoding relation direction and names.
- Plan realization: Recognized entities are replaced with unique tokens so the NMT system can copy them rather than generate their surface forms.
- Training details: Training uses a standard NMT setup with copy attention and pretrained GloVe embeddings initialized for plan relations and reference-text tokens.
- Generation details: After generation, entity tokens are replaced by input-graph strings, while dates and numbers with units receive postprocessing normalization.
5 Experimental Setup
The experiments evaluate the planning-realization approach on WebNLG against established grammar-based and neural systems, including a stronger neural baseline. The recovered plans cover 76% of reference texts.
- Dataset: WebNLG maps RDF-triplet sets containing up to seven triplets to reference texts and includes seen and unseen test inputs.
- Dataset: The authors recover consistent plans for 76% of reference texts, yielding 13,828 plan-text pairs from 18,102 RDF-text pairs.
- Compared systems: Comparisons include Melbourne, the best automatic-evaluation WebNLG submission, and UPF-FORGe, the best human-evaluation submission.
- Compared systems: StrongNeural is an end-to-end baseline combining a set encoder, attentive LSTM decoder, copy attention, neural checklist, and entity dropout.The entity-dropout and checklist components distinguish it from previous systems.
6 Experiments and Results
Experiments on WebNLG show that BestPlan matches neural systems on automatic metrics and fluency while substantially improving semantic faithfulness. Its realizations generally follow selected plans, supporting structural control and diverse outputs.
- 6.1 Automatic Metrics: BestPlan is competitive with StrongNeural across BLEU, Meteor, ROUGEL and CIDEr, while both outperform all WebNLG participating systems.The comparison covers the entire test set and its seen and unseen portions.
- 6.2 Manual Evaluation: BestPlan reduces all measured semantic-faithfulness error types versus StrongNeural by 85%, 56% and 90%.The evaluation concerns 440 RDF triplets from 139 input sets in the seen test set.
- 6.2 Manual Evaluation: BestPlan is as fluent as StrongNeural, surpasses UPF-FORGe, and scores below human reference texts.The authors attribute retained fluency to the NMT realization component; the reference gap is expected because BestPlan does not produce referring expressions.
- 6.3 Plan Realization Consistency: For decreasingly probable plans, the realizer performs worse at expressing all entities, but realizations that include all entities follow the requested order.This consistency holds for both seen and unseen test sets.
- 6.3 Plan Realization Consistency: Realizing random plans from the top 10% maintains decent automatic-metric quality with a limited performance drop and stable scores across random seeds.Three random seeds generate the RandomPlan-1/2/3 systems.
7 Related Work
Related work frames text planning as a longstanding NLG component and positions this approach among methods for ordering, coverage, document planning, and controllable generation. The paper also acknowledges that its current plan-selection procedure is simple and that sentence order is heuristic.
- Text Planning: Classic NLG work explicitly models text planning, including generation and ranking of multiple sentence-plan trees.The paper’s approach similarly uses reference plans but lacks internal rankings among them.
- Text Planning: The paper determines sentence order with a fact-count splitting heuristic rather than content-based ordering, leaving room for probabilistic ordering models.The authors cite Lapata’s model as a related approach that correlated well with human ordering.
- Coverage: Coverage tracking and copy mechanisms address over-generation and missing facts, making them complementary to explicit planning.The paper describes these methods as effective for ensuring coverage and reducing over-generated facts.
- Document Planning: Prior neural work also explores explicit document planning for more complex tasks such as RotoWire.Puduppully et al. model document planning using attention.
- Controllable Generation: Controllable-generation research manipulates attributes such as sentiment, whereas this work controls content and expression through the sentence plan.The paper relates this focus to prior work on manipulating sentence plans.
8 Conclusion
The plan-based system matches a strong end-to-end neural system on automatic metrics and fluency, while substantially improving faithfulness. Explicit planning also supports user control and diverse outputs.
- The plan-based system performs on par with a strong end-to-end neural system on automatic evaluation metrics.
- The plan-based system substantially outperforms the end-to-end system in faithfulness to the input.
- The planning stage enables explicit user control and diverse sentence generation as future work.
A Diverse Outputs
The model generates diverse realizations by sampling many ranked plans for graphs with four, five, and six edges. Plans encode entities, bracketing, and sentence splits, while generated texts mark entity realizations.
- The model demonstrates diverse outputs for graphs with 4, 5, and 6 edges.For each graph, 25 examples cover the top 10% of plans.
- Plan visualizations mark entities, bracketing, and sentence splits, while generated texts underline corresponding entities.
- The displayed examples cover graphs from the seen part of the test set.
- For the 4-edge graph, 25 random plans are drawn from 1,295 possible plans and paired with generated realizations.
- For the 6-edge graph, 25 random plans are drawn from 171,024 possible plans and paired with generated realizations.
B.1 Fluency Evaluation by Crowd
Fluency was evaluated through pairwise comparisons of system outputs in the original WebNLG manual evaluation setup. Mechanical-Turk workers selected which text was more fluent or whether both were equally good or bad.
- Fluency evaluation used RDF sets from the original WebNLG manual evaluation setup.
- Mechanical-Turk workers compared two system texts shown in random order alongside the RDF triplets.
- Workers answered which text reads more fluently, choosing Text 1, Text 2, or Both are equally good or bad.
- Each hit used three qualified workers paid $0.08, with qualification thresholds of over 98% approval and 1,000 approved hits.
- Fine-grained semantic faithfulness was separately annotated for StrongNeural and BestPlan system outputs.
C Training Parameters
The realization model uses OpenNMT with pretrained GloVe.6B embeddings and default parameters, except for enabling copy attention.
- The realization model uses the OpenNMT toolkit with pretrained GloVe.6B word embeddings.
- The model uses OpenNMT default parameters except for the -copy attn flag.
- The passage states that these implementation choices correspond to specified parameter values.