PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot NLG with Pre-Trained Language Model

Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, William Yang Wang

arXiv:1904.09521v3cs.CL

TL;DR

Neural NLG from structured data is data-hungry, motivating few-shot generation with limited labeled examples. The paper combines learned content selection with prior language-model knowledge and reports strong, cross-domain performance, including an average improvement of over 8.0 BLEU points over the strongest baseline.

  • Problem

    Neural-based NLG from structured data requires large training datasets, creating substantial manual data-curation overhead for real-world adoption.

  • Method

    The approach learns content selection from tables while using a pre-trained language model as prior knowledge for fluent sentence composition.

  • Results

    With 200 training examples across multiple domains, the approach improves over the strongest baseline by an average of over 8.0 BLEU points.

  • Takeaways & Limitations

    Few-shot NLG can achieve reasonable performance across domains while reducing reliance on extensive labeled data.

  • Takeaways & Limitations

    The work excludes target-text information not present in the input infobox and filters out examples containing rare words outside the infobox.

Abstract

from arXiv · show

Neural-based end-to-end approaches to natural language generation (NLG) from structured data or knowledge are data-hungry, making their adoption for real-world applications difficult with limited data. In this work, we propose the new task of \textit{few-shot natural language generation}. Motivated by how humans tend to summarize tabular data, we propose a simple yet effective approach and show that it not only demonstrates strong performance but also provides good generalization across domains. The design of the model architecture is based on two aspects: content selection from input data and language modeling to compose coherent sentences, which can be acquired from prior knowledge. With just 200 training examples, across multiple domains, we show that our approach achieves very reasonable performances and outperforms the strongest baseline by an average of over 8.0 BLEU points improvement. Our code and data can be found at \url{https://github.com/czyssrs/Few-Shot-NLG}

1 Introduction

The paper frames few-shot NLG as generating satisfactory text from structured data with only tens to hundreds of labeled examples, addressing the data demands of neural NLG. Its approach separates table-content selection from language composition and uses pre-trained language knowledge to generalize across domains.

  • Motivation: Few-shot NLG asks neural systems to produce satisfactory text from only 50–200 labeled instances, addressing the large annotation burden of conventional data-driven NLG.The paper gives BLEU ≥20 as an example of satisfactory output.
  • Approach: The approach separates factual content selection and copying from sentence composition, treating these as distinct skills for table-to-text generation.The paper argues that content selection can be learned from a handful of tables, while sentence composition is domain-independent.
  • Approach: A pre-trained language model supplies prior language knowledge, while a switch policy enables the model to alternate between copying table content and generating text.The architecture uses a pre-trained language model as the generator and keeps the newly learned parameter space small.
  • Contributions: The paper introduces a multi-domain table-to-text dataset to study few-shot NLG algorithms.The training and test instances are crawled from Wikipedia.
  • Results: The method improves the baseline by an average of over 8.0 BLEU across domains and achieves reasonable performance with just 200 training instances.The paper presents the reduction of annotation effort and cross-domain evaluation as central contributions.

2 Related Work

Earlier NLG systems used pipelines with templates and hand-engineered features, while later neural methods unified content selection and realization but required massive training data. This work is positioned in contrast to approaches needing large paired or target-side datasets.

  • NLG from Structured Data: Traditional NLG decomposed generation into content selection, planning, and surface realization, relying heavily on templates and hand-engineered features.Later pipeline work modeled context selection and surface realization in a unified framework.
  • Neural NLG: Neural end-to-end systems jointly model context selection and surface realization, achieving strong benchmark performance while depending on massive training datasets.The cited benchmarks include E2E, WebNLG, and WIKIBIO.
  • Few-shot Setting: Unlike zero-shot question-generation work requiring 1,000 paired examples and large target-side corpora, this setting uses tens to hundreds of paired examples without target examples.The paper characterizes this setting as closer to real-world conditions where large target-side references are difficult to obtain.
  • Pre-training: Language-model pre-training provides downstream tasks with rich prior knowledge from large-scale open-domain corpora.The related work discusses word, sentence, and language-model-based pre-training, including BERT and GPT-2.

3 Method

The method separates table content selection and copying from language modeling, using a switch policy and a pre-trained language model to generate coherent text from semi-structured data with few examples.

  • 3.1 Problem Formulation: The input is a semi-structured table of attribute-value pairs, where values are word sequences annotated with attribute names and within-value positions.The target is a natural-language description generated from this table using only a handful of training instances.
  • 3.2 Base Framework with Switch Policy: A switch policy separates table content selection/copying from language-model-based generation, choosing between vocabulary generation and copying table values at each time step.The model maintains a soft copy probability, with attention weights defining the copying distribution.
  • 3.2 Base Framework with Switch Policy: The model explicitly supervises copying by matching target text with table values and adding a copy loss that maximizes copy probability at matched positions.The loss combines the original generation loss with a weighted copy loss, and the decoder input includes matched attribute and position information.
  • 3.3 Pre-Trained LM as Generator: A pre-trained language model serves as the generator, while its word embeddings remain fixed to support generalization to tokens unseen in the few-shot training data.The architecture uses GPT-2, fine-tuning generator parameters while learning the encoder and attention components from scratch.
  • 3.3 Pre-Trained LM as Generator: The architecture applies a larger copy-loss weight because the fine-tuned generator and newly learned encoder-attention components begin with different parameter geometries.This stronger signal is intended to teach the model to copy facts from the input table.

4 Experiment

Experiments evaluate the method across multi-domain few-shot table-to-text settings, compare architectural baselines, and assess factuality and naturalness with human judgments.

  • 4.1 Experimental Setup: The evaluation uses Humans, Books, and Songs domains, with datasets collected from Wikipedia and experiments varying the number of training instances.The Books and Songs collections contain 23,651 and 39,450 instances, respectively; Humans comes from the original WIKIBIO dataset.
  • 4.2 Results and Analysis: Over 8.0 BLEU points of average improvement comes from adding the pre-trained language model to the switch-based baseline.The switch policy first improves average BLEU by over 10.0 points, while the language-model augmentation adds another average improvement of over 8.0 points.
  • 4.2 Results and Analysis: Table 3 measures the copy-loss ablation on Humans using BLEU-4 and reports an average improvement of over 4.0 BLEU points.The copy loss provides a stronger signal for learning to copy from the input table.
  • 4.2 Results and Analysis: Compared with Pivot, the method achieves better performance without requiring additional large-scale target-side data.The comparison uses the original WIKIBIO dataset to preserve the target-side-example setting of the Pivot model.
  • Human Evaluation: Human evaluation assesses factual correctness through supporting and contradicting facts and assesses language naturalness through pairwise comparisons.The study evaluates 500 samples, assigns each unit to three workers, and finds significant improvement over the strongest baseline across measures.

5 Conclusion

The paper concludes that few-shot NLG can use language-modeling priors to achieve strong performance across domains with a simple approach.

  • 5 Conclusion: The approach targets reduced manual data curation for structured-data generation and may extend to knowledge graphs and SQL queries.The conclusion describes the approach as simple and easy to implement while achieving strong performance across various domains.

Appendix A. Implementation Details

Implementation uses Adam optimization and specifies training, copy-loss, and embedding settings for reproducibility.

  • Appendix A. Implementation Details: The implementation uses Adam with learning rate 0.0003, mini-batches of 40, and copy-loss weight λ set to 0.7.Position embeddings have dimension 5, and multiword attribute names use averaged word embeddings.

Appendix B. ROUGE-4 Results

The appendix reports ROUGE-4 results for the Humans and Books domains, following automatic evaluation with BLEU-4 and ROUGE-4 F-measure.

  • The evaluation uses BLEU-4 and ROUGE-4 F-measure as automatic metrics.
  • ROUGE-4 results are reported for the Humans domain in Table 6.
  • ROUGE-4 results are reported for the Books domain in Table 7.

Appendix C. Human Evaluation Details

The human evaluations assess factual correctness and language naturalness using outputs trained on 200 Humans-domain examples. They use multiple workers per item and statistical testing to compare generation methods.

  • Evaluation Setup: Human evaluations use 500 randomly sampled test instances and outputs trained with 200 Humans-domain examples.Each evaluation unit is assigned to three workers to reduce human-response variance.
  • Factual Correctness: Factual correctness is measured by counting facts supported, contradicted, or missing relative to the input table.Workers see both the table and generated text, and method-level averages are computed.
  • Language Naturalness: Language naturalness is evaluated through pairwise judgments of grammaticality and fluency without showing workers the input table.Workers select the better text or “Tied,” based only on language naturalness.
  • Statistical Analysis: Statistical significance is tested with Tukey HSD post-hoc analysis of an ANOVA across factuality and naturalness measures.Worker responses are modeled with method and worker identity as independent variables.
Loading 1904.09521v3…