PaperlayerView exact revision on arXiv

Source-linked AI summary

PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation

Zhe Hu, Hou Pong Chan, Jiachen Liu, Xinyan Xiao, Hua Wu, Lifu Huang

arXiv:2203.09100v1cs.CL

TL;DR

Pretrained language models generate fluent text but often lack coherent content planning for long-form, multi-sentence generation. PLANET dynamically unifies sentence-level planning and surface realization with latent representations, bag-of-words supervision, and coherence-based contrastive learning. Across counter-argument and opinion-article generation, it outperforms strong baselines and produces more coherent, content-rich texts in automatic and human evaluations.

  • Problem

    Pretrained language models remain prone to incoherence in long-form generation, where content must be selected, ordered, and realized across sentences.

    Source: §1 · p. 1

  • Method

    PLANET uses autoregressive self-attention, latent sentence representations grounded by bag-of-words plans, and coherence-based contrastive learning for unified content planning and realization.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    PLANET outperforms strong baselines on two long-form opinion generation tasks in automatic and human evaluations, producing more coherent texts with richer contents.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The framework effectively organizes and reuses guidance keyphrases as coherent talking points in generated opinion texts.

    Source: §5.3 · p. 9

    Results are consistently better on OpinionGen than ArgGen, possibly because ArgGen contains more colloquial and informal social-network language that makes implicit coherence harder to learn.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Despite recent progress of pre-trained language models on generating fluent text, existing methods still suffer from incoherence problems in long-form text generation tasks that require proper content control and planning to form a coherent high-level logical flow. In this work, we propose PLANET, a novel generation framework leveraging autoregressive self-attention mechanism to conduct content planning and surface realization dynamically. To guide the generation of output sentences, our framework enriches the Transformer decoder with latent representations to maintain sentence-level semantic plans grounded by bag-of-words. Moreover, we introduce a new coherence-based contrastive learning objective to further improve the coherence of output. Extensive experiments are conducted on two challenging long-form text generation tasks including counterargument generation and opinion article generation. Both automatic and human evaluations show that our method significantly outperforms strong baselines and generates more coherent texts with richer contents.

1 Introduction

Long-form opinion generation requires sentence-level content selection, ordering, and realization, yet fluent pretrained models often produce incoherent multi-sentence texts. PLANET unifies dynamic planning and surface realization with latent sentence plans, bag-of-words supervision, and coherence-based contrastive learning.

1 Introduction

  • Long-form opinion generation must select and order what to say while also deciding how to express those plans coherently across sentences.The task uses statements and unordered keyphrases as guidance for producing coherent multi-sentence opinions.Source: §1 · p. 1
  • BART can include guidance keyphrases yet produce incorrect keyphrase usage, wrong stance, and weak transitions between sentences.The counter-argument example contrasts these failures with a human writer’s sentence-by-sentence planning and keyphrase selection.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Earlier planning methods either use global latent plans without fine-grained sentence planning or separate planning from realization, risking cascading errors.PLANET addresses both limitations through unified autoregressive self-attention.Source: §1 · p. 2
  • PLANET dynamically performs sentence-level content planning and surface realization in autoregressive Transformers, using latent representations to guide each sentence’s semantic content and wording.The framework is trained end-to-end and can be applied to pretrained autoregressive Transformers.Source: §1 · p. 2
  • PLANET grounds each sentence’s latent semantic plan with a sentence-level bag-of-words auxiliary task and adds coherence-based contrastive learning with negative-sample strategies.These objectives supervise lexical semantics and help distinguish coherent from incoherent targets.Source: §1 · p. 2
  • PLANET outperforms strong baselines in automatic and human evaluations on counter-argument generation and opinion article generation, producing more coherent and content-rich texts.Human evaluations also indicate that it leverages guidance keyphrases appropriately on both datasets.Source: §1 · p. 2

2 Related Work

Related work addresses long-form generation through hierarchical planning, separate planning and realization modules, structured information, and controllable generation. PLANET differs by integrating sentence-level planning and surface realization dynamically in pretrained autoregressive Transformers.

Text Planning for Neural Generation

  • Earlier neural text-planning methods use hierarchical decoders, but architectural modifications make them difficult to apply to pretrained models.Other separate planning modules can disconnect content planning from surface realization and produce undesired outputs.Source: §2 · pp. 2–3
  • Separate planning-and-realization systems often omit sentence-level planning, disconnect the components, or fail to model interactions among content items.PLANET instead introduces high-level latent representations for target sentences within pretrained autoregressive Transformers.Source: §2 · p. 3

Coherent Long-form Text Generation

  • Coherence research covers stories, paragraphs, infilling, conversations, news, discourse structure, and auxiliary similarity or order objectives.PLANET focuses on opinion text generation requiring credible talking points and rigorous reasoning, combining dynamic planning with coherence-based contrastive learning.Source: §2 · p. 3

Controllable Text Generation

  • Controllable-generation studies steer text through sentiment, style, syntax, or keywords, whereas PLANET organizes topical keyphrases as talking points for long-form opinion text.Its goal is to properly organize and reflect those keyphrases in coherent outputs.Source: §2 · p. 3

3 Our PLANET Framework

PLANET unifies dynamic sentence-level content planning and surface realization in an autoregressive Transformer, using latent semantic plans, keyphrase selection, sentence-level bag-of-words supervision, and coherence-based contrastive learning.

3.1 Framework Overview

  • PLANET dynamically performs content planning and surface realization through autoregressive self-attention in a unified framework.The decoder creates sentence-level latent representations and generates sentence words within the same autoregressive process.Source: §3.1 · p. 3

3.2 Latent Representation Learning

  • For each target sentence, latent representations summarize semantic information and guide surface realization while attending to prior sentence representations and words.The approach inserts [SN] tokens whose decoder hidden states represent target-sentence plans, allowing planning and realization to occur simultaneously and remain end-to-end trainable.Source: §3.2 · pp. 3–4

3.3 Content Selection

  • At each decoding step, decoder states attend to a keyphrase memory bank, integrate selected keyphrase context through a residual layer, and estimate output-word probabilities.The memory bank is built from encoder representations of keyphrase boundary tokens.Source: §3.3 · pp. 4–5

3.4 Sentence-level Bag-of-words Planning

  • Sentence-level bag-of-words supervision grounds each latent representation in a target sentence’s global semantic plan and guides word generation.The model predicts a vocabulary distribution for each sentence and conditions autoregressive word probabilities on the corresponding latent representation.Source: §3.4 · p. 5

3.5 Coherence-based Contrastive Learning

  • Coherence-based contrastive learning trains the model to score coherent targets above incoherent negatives constructed by shuffling, replacing, differentiating, or masking target content.The original target is treated as positive, while four negative-sample strategies target sentence order, content organization, input mismatch, and keyphrase-related word corruption.Source: §3.5 · p. 5

3.6 Training Objective

  • The joint training objective combines token-generation cross-entropy, sentence-level bag-of-words planning, and contrastive losses.The losses are combined as L = LGEN + αLBOW + βLCL, with α and β tuned as hyperparameters.Source: §3.6 · pp. 5–6

4 Experimental Setups

PLANET is evaluated on counter-argument and opinion-article generation using guidance keyphrases, BART-based baselines, and controlled training and decoding settings.

4.1 Tasks and Datasets

  • The experiments cover ArgGen, which generates counter-arguments from propositions, and OpinionGen, which generates opinion articles from titles.ArgGen uses Reddit r/ChangeMyView arguments and extracted guidance keyphrases, while OpinionGen uses New York Times Opinion articles and the same keyphrase extraction method.Source: §4.1 · p. 6

4.2 Baselines and Comparisons

  • The comparison includes retrieval, hierarchical planning, BART fine-tuning with keyphrases, global planning, and a two-stage planner-generator.SEPPLAN separately predicts ordered sentence-level keyphrase plans and then generates outputs from those plans.Source: §4.2 · p. 6

4.3 Training and Decoding Details

  • All experiments use BART-base, truncate inputs and targets to 256 tokens, and train with sentence-level bag-of-words and contrastive objectives.The loss weights are α=0.2 for ArgGen and 0.3 for OpinionGen, β=0.2, and the contrastive margin is 0.5.Source: §4.3 · p. 6
  • Decoding uses nucleus sampling with threshold 0.9 and maximum generation lengths of 150 steps for ArgGen and 200 for OpinionGen.The model is optimized with AdamW.Source: §4.3 · p. 6

5 Results and Analysis

PLANET and its variants outperform the baselines across automatic and human evaluations, with gains in coherence, content richness, and keyphrase use; qualitative examples show fewer incoherent outputs.

5.1 Automatic Results

  • PLANET without contrastive learning consistently outperforms all baseline methods, including FULLSEQ2SEQ, SSPLANER, HIERPLAN, and SEPPLAN.The comparison supports dynamic content planning and end-to-end training over separately planned generation pipelines.Source: §5.1 · pp. 6–7, Table 2 · p. 7
  • Removing content selection or bag-of-words planning decreases performance, while adding contrastive loss improves automatic results, especially BLEU.The authors link these gains to more effective content planning and more precise use of keyphrases.Source: §5.1 · p. 7
  • PLANET generates more distinct n-grams than the baselines among generative methods, and adding contrastive loss produces still more unique n-grams.RETRIEVAL has the highest distinct n-gram results overall because it returns human-written texts.Source: §5.1 · p. 7, Figure 5 · p. 7
  • PLANET achieves the highest coherence scores on both datasets, while PLANET without contrastive learning remains more coherent than FULLSEQ2SEQ and SSPLANER.SEPPLAN has the lowest coherence scores, and the authors associate this with decoupled planning and decoding stages.Source: §5.1 · p. 7, Figure 6 · p. 7
  • Removing any negative-sample construction strategy degrades coherence, with REPLACE having the largest effect and DIFFERENT the smallest.The authors hypothesize that sentence replacement disrupts logical flow, whereas DIFFERENT emphasizes topical relatedness rather than within-output logic.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 Human Evaluation

  • Human evaluations show both PLANET variants outperform FULLSEQ2SEQ on relatedness, coherence, and content richness, while adding a contrastive objective further improves quality and makes outputs more likely to be top-ranked.Judges score outputs from 1 (worst) to 5 (best) on the three aspects and select top-ranked results based on overall quality; 50 random samples are selected from each task.Source: §5.2 · p. 8
  • PLANET covers more guidance keyphrases than FULLSEQ2SEQ and SSPLANER, and human judges rate its keyphrase usage more favorably than baseline methods.On OpinionGen, the contrastive model mentions fewer keyphrases but receives higher human keyphrase-usage scores.Source: §5.2 · pp. 8–9, Figure 7 · p. 8

5.3 Sample Outputs and Discussions

  • Qualitative examples show PLANET organizing and reusing guidance keyphrases coherently, whereas FULLSEQ2SEQ exhibits repetition and inconsistent stance.The predicted sentence-level bag-of-words captures most salient target content words, though it remains below human-written texts.Source: §5.3 · p. 9, Figure 8 · p. 9

6 Conclusion

PLANET unifies dynamic content planning and surface realization in autoregressive Transformers through latent sentence plans grounded in bag-of-words representations. A coherence-based contrastive objective further improves output coherence, and experiments on two opinion-generation tasks yield higher-quality, richer outputs.

6 Conclusion

  • PLANET dynamically performs content planning and surface realization with autoregressive self-attention and sentence-level latent representations grounded by bag-of-words.The latent representations encode each target sentence’s overall semantic plan.Source: §6 · p. 9
  • A coherence-based contrastive objective with multiple negative-sample strategies is introduced to improve the coherence of generated text.The framework is evaluated on counter-argument and opinion-article generation tasks.Source: §6 · p. 9
  • Experiments demonstrate higher-quality outputs with better coherence and content richness on both opinion text generation tasks.Source: §6 · p. 9

Ethics Statement

The paper warns that open-ended opinion generation and heterogeneous web pretraining can produce fabricated or harmful content, requiring careful real-world use.

Ethics Statement

  • PLANET may generate fabricated and potentially harmful content because of pretraining biases and the open-ended nature of opinion-generation tasks.The authors urge users to examine ethical effects and apply the system cautiously in real-world settings.Source: Ethics Statement · p. 10

A.1 Additional Experimental Results

  • Additional automatic results report BLEU-3 and ROUGE-L recall for both tasks in Tables 5 and 6.Source: §A.1 · p. 13

A.2 Training and Decoding Details

The experiments use a BART-base implementation with dynamic sentence control during decoding and a separately trained coherence model for contrastive learning. Training and decoding details specify resource limits, sampling settings, and coherence-model validation.

A.2 Training and Decoding Details

  • BART-base is trained with sentence-start [SN] tokens, 185M parameters, batch size 8, and up to 15 or 18 epochs depending on contrastive training.Inputs and targets are truncated to 256 tokens, and the best checkpoint is selected by validation loss.Source: §A.2 · p. 13
  • Nucleus sampling uses k = 10 and p = 0.9, with maximum generation lengths of 150 ArgGen steps and 200 OpinionGen steps.Outputs are decoded three times for averaged automatic evaluation, and generation stops at <EOS>.Source: §A.2 · p. 13
  • PLANET dynamically plans and realizes each sentence by enforcing [SN] boundaries, allowing the model to decide how many sentences to generate during inference.The model generally produces [SN] automatically, while decoding enforces it after detected sentence endings.Source: §A.2 · p. 13
  • SEPPLAN provides a two-stage baseline in which one BART generates ordered sentence-level keyphrase plans and another generates text from those plans.Ground-truth plans train the generator, whereas predicted plans are used during inference.Source: §A.2 · pp. 13–14

A.3 Training Details for Coherence Model

  • The coherence model fine-tunes BERT on whole texts, using shuffled-sentence negatives and hinge loss; its test accuracy is 94.3% on OpinionGen and 73.0% on ArgGen.Whole-text scoring is intended to capture global rather than only adjacent-sentence coherence.Source: §A.3 · p. 14

B Details for Human Evaluation

Human evaluation uses anonymized, shuffled model outputs from 55 random samples per task and assesses multiple dimensions of generation quality. Agreement is generally positive across the evaluated aspects.

B Details for Human Evaluation

  • 55 random samples per task are evaluated, with the first 5 used only for calibration and model identities anonymized.Outputs are shuffled before being shown to annotators.Source: B Details for Human Evaluation · p. 14
  • Human judges assess relatedness, coherence, content richness, and overall ranking using aspect-specific guidelines.The criteria cover input consistency, logical flow, substantive details, and whether an output ranks among the best candidates.Source: B Details for Human Evaluation · p. 14
  • Krippendorff’s α is at least 0.34 for every aspect on both datasets, indicating general consensus among judges.The agreement values are reported in Table 7.Source: B Details for Human Evaluation · p. 14, Table 7 · p. 14

C Discussions on Limitations and Future Directions

The paper reports remaining gaps against human writing and identifies scope boundaries around guidance availability, supervision signals, and the studied generation setting. It proposes broader supervision, pretraining, and guidance sources as future directions.

C Discussions on Limitations and Future Directions

  • PLANET still has a gap compared with human-written outputs despite improving coherence and content richness over strong baselines.The limitation is stated for the reported long-form generation results.Source: C Discussions on Limitations and… · p. 14
  • The method assumes keyphrases are available as guidance signals at test time, leaving unguided scenarios outside the paper’s scope.The paper suggests retrieval-based or knowledge-enhanced modules to obtain keyphrases when unavailable.Source: C Discussions on Limitations and… · p. 14
  • Future work could apply dynamic planning during pretraining or post-pretraining, using automatically constructed keyphrase and bag-of-words labels without additional annotation.The authors suggest leveraging massive pretraining data across domains.Source: C Discussions on Limitations and… · p. 15
  • Bag-of-words supervision captures overall semantics, while discourse structures, entity usage, and other signals could address additional dimensions of coherence.The paper names topical relatedness, causal relationships, temporal ordering, and discourse structures as distinct coherence aspects.Source: C Discussions on Limitations and… · p. 15
  • Future work could replace or supplement keyphrase guidance with heterogeneous sources such as structured knowledge and commonsense information.This direction is proposed to further improve output quality.Source: C Discussions on Limitations and… · p. 15

D Additional Sample Outputs

The appendix provides additional ArgGen and OpinionGen examples, displaying sentence-level bag-of-words guidance alongside generated text. Highlighted phrases show how outputs relate to the guidance keyphrases.

D Additional Sample Outputs

Loading 2203.09100v1…