PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Pipeline for Zero-Shot Data-to-Text Generation

Zdeněk Kasner, Ondřej Dušek

arXiv:2203.16279v1cs.CL

TL;DR

Fine-tuning PLMs on D2T data can overfit to a benchmark, while gathering human references is costly. The paper instead uses PLMs trained on general-domain text operations to generate descriptions from RDF triples without D2T training data. It reports gains over simple baselines and fluency comparable to older supervised systems, though not state-of-the-art quality.

  • Problem

    Can PLMs generate D2T text without domain-specific fine-tuning, which risks benchmark overfitting and requires costly human references?

    Source: §1 · p. 1

  • Method

    The system templates RDF triples as sentences, then orders, aggregates, and paragraph-compresses them using PLMs trained on WikiFluent.

    Source: §1 · p. 2, §3.1 · p. 3, §3.4 · p. 4

  • Results

    The pipeline improves on simple baselines and matches older supervised systems on automatic fluency metrics, but does not match state-of-the-art quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Avoiding noisy in-domain references yields more semantically consistent outputs, and analyses suggest a way to prevent omissions and hallucinations common in few-shot approaches.

    Source: §1 · p. 2, §9 · p. 9

    Outputs are not guaranteed factually correct; WebNLG outputs also merge unrelated facts, producing semantic errors.

    Source: §7.2 · p. 8, §10 · p. 10

Abstract

from arXiv · show

In data-to-text (D2T) generation, training on in-domain data leads to overfitting to the data representation and repeating training data noise. We examine how to avoid finetuning pretrained language models (PLMs) on D2T generation datasets while still taking advantage of surface realization capabilities of PLMs. Inspired by pipeline approaches, we propose to generate text by transforming single-item descriptions with a sequence of modules trained on general-domain text-based operations: ordering, aggregation, and paragraph compression. We train PLMs for performing these operations on a synthetic corpus WikiFluent which we build from English Wikipedia. Our experiments on two major triple-to-text datasets -- WebNLG and E2E -- show that our approach enables D2T generation from RDF triples in zero-shot settings.

1 Introduction

The introduction proposes zero-shot D2T generation with PLMs operating on natural-language text, addressing concerns about in-domain fine-tuning. It reports gains over simple baselines and fluency comparable to older supervised systems, while surveying related pipeline and text-operation tasks.

1 Introduction

  • Fine-tuning on human-written D2T references is standard, but benchmark overfitting and costly reference collection motivate alternatives; few-shot robustness is also questioned.The introduction also notes that limited, noisy few-shot samples may increase hallucinations and omissions.Source: §1 · p. 1
  • The zero-shot pipeline improves on simple baselines and matches older supervised systems on automatic fluency metrics, but does not match state-of-the-art quality.Semantic-accuracy metrics and manual analysis suggest it may help prevent omissions and hallucinations common in few-shot approaches.Source: §1 · p. 2
  • WikiFluent supplies 934k synthetic examples based on English Wikipedia for training the general-domain text operations.The examples pair synthesized template-like sentences with the meaning of original paragraphs.Source: §1 · p. 2

2 Related Work

  • Unlike prior D2T planners that represent items with dataset predicates or keys, this approach plans over natural-language sentences in a zero-shot pipeline.Its aggregation setup inserts sentence-boundary delimiters into an already ordered sequence of facts.Source: §2 · pp. 2–3
  • Paragraph compression is introduced as the pipeline’s final task, combining sentence fusion, rephrasing, and coreference resolution while aiming to preserve all facts.Unlike summarization or simplification, it targets complete semantics across multiple sentences in a paragraph.Source: §2 · p. 3

3 Method

The pipeline converts RDF triples into natural-language facts, orders them, marks which neighbors to aggregate, and compresses the marked sequence into a description.

3.1 Transforming Triples to Facts

  • Each triple becomes a natural-language fact, preparing it for later text-to-text operations while introducing predicate semantics.A fact is a single sentence describing its triple; a simple predicate-specific template can perform the transformation.Source: §3.1 · p. 3

3.2 Ordering the Facts

  • Because input facts may be randomly ordered, an ordering model arranges them for coherence, letting later modules focus on neighboring sentences.The paper gives grouping birth date and birthplace facts before occupation as an example.Source: §3.2 · p. 3

3.3 Aggregating the Facts

  • Given ordered facts, the aggregation model predicts sentence delimiters: 1 keeps neighbors separate, while 0 marks them for fusion.Facts likely to be mentioned together can be aggregated into one sentence.Source: §3.3 · p. 3

3.4 Paragraph Compression

  • Paragraph compression takes the ordered facts and delimiters and generates the final description, fusing marked sentences and rephrasing for fluency while preserving semantics.Because the input is already ordered, edits are typically minor.Source: §3.4 · p. 4

4 WIKIFLUENT Corpus

WIKIFLUENT supplies training examples for the pipeline modules, pairing synthesized simple sentences with human-written Wikipedia paragraphs. Its construction uses splitting, coreference replacement, and semantic filtering.

4 WIKIFLUENT Corpus

  • The corpus aims to pair template-like simple sentences with fluent, meaning-preserving text, using synthesized sentences as inputs and original paragraphs as ground truth.Each paragraph sentence is split and the synthesized sentences are resolved for coreference.Source: §4 · p. 4, Figure 2 · p. 4

4.1 Data Source

  • WIKIFLUENT draws on 934k first paragraphs from English Wikipedia, selected across length ranges to balance input lengths.The source paragraphs are concise, fact-based descriptions; the selection excludes lists, disambiguations, and repeated or malformed paragraphs.Source: §4.1 · p. 4

4.2 Split-and-Rephrase

  • For split-and-rephrase, a model trained on WikiSplit is applied to each sentence with 0–2 randomly chosen recursive calls; unsplittable sentences are retained unchanged.WikiSplit contains human-made sentence splits from Wikipedia edit history.Source: §4.2 · p. 4

4.3 Coreference Replacement

  • Coreference resolution replaces referring expressions with antecedents in synthesized inputs, while original paragraphs retain them, implicitly training compression to generate referring expressions.Source: §4.3 · p. 5

4.4 Filtering

  • A pretrained RoBERTa model trained on MultiNLI checks synthesized sentences for omissions and hallucinations; retaining examples passing both checks yields a filtered WIKIFLUENT corpus of 714k examples, approximately 75% of the original size.Source: §4.4 · p. 5

5 Implementation

The implementation combines template transformations with neural pipeline modules trained on WIKIFLUENT.

5 Implementation

  • The implementation uses simple template transformations and neural models trained on WIKIFLUENT for the pipeline modules.Source: §5 · p. 5

5.1 Templates

  • Each predicate receives a single triple template that transforms its subject, predicate, and object into a fact sentence.For example, an instrument predicate can yield “<s> plays <o>.”Source: §5.1 · p. 5
  • The authors use simple hand-crafted templates as an efficient way to introduce domain knowledge about predicates.They choose this simpler approach to showcase their pipeline independently of template-generator quality.Source: §5.1 · p. 5

WebNLG

The implementation uses neural models trained on WIKIFLUENT for ordering, aggregation, and paragraph compression. For WebNLG and E2E, predicate templates provide the initial facts.

WebNLG

  • WebNLG and E2E predicate templates turn triples into facts; simpler templates may yield less fluent outputs but reduce manual work and ease control of subsequent inputs.Source: WebNLG · p. 5, Table 1 · p. 5

5.2 Ordering Model

  • The ordering model encodes the facts and uses a pointer network to select their sequence; training shuffles synthesized sentences and teaches the model to restore their original order.The model uses a pretrained BART-base extended with a pointer network.Source: §5.2 · pp. 5–6

5.3 Aggregation Model

  • The aggregation model classifies separators between ordered facts, marking originally aggregated neighboring sentences with 0 and other boundaries with 1.Source: §5.3 · p. 6

5.4 Paragraph Compression Model

  • The paragraph compression model is fine-tuned on WIKIFLUENT with delimiters marking non-aggregated boundaries, then generates text from the concatenated sentences.The delimiters are represented with a special <sep> token.Source: §5.4 · p. 6

6 Experiments

The experiments apply the WikiFluent-trained pipeline modules to WebNLG and E2E without finetuning on those datasets, comparing pipeline variants that assign ordering and aggregation to different stages.

6 Experiments

  • The evaluation applies the modules without finetuning to WebNLG RDF triples and E2E attribute-value pairs converted into RDF triples.WebNLG uses version 1.4 and hand-crafted templates for all 354 predicates; E2E uses a manually created template for each of its eight attributes.Source: §6 · p. 6
  • The paragraph-compression ablation varies whether its input facts are ordered and delimited, leaving aggregation and ordering implicit in progressively more integrated variants.PC receives ordered facts with delimiters; PC+AGG receives ordered facts without delimiters; PC+ORD+AGG receives randomly ordered facts without delimiters.Source: §6 · pp. 6–7
  • The three-stage pipeline uses separate ordering, aggregation, and paragraph-compression models; two-stage combines aggregation with compression, while one-stage combines all three operations.All variants use paragraph-compression models trained on either the full or filtered WikiFluent corpus.Source: §6 · p. 7

7 Evaluation and Discussion

The evaluation finds that the zero-shot systems beat COPY but remain below state-of-the-art supervised systems. Manual and module-level assessments identify factual-merging errors as a key weakness while showing that the compression model generally follows its content plan.

7.1 Automatic Metrics

  • The systems consistently outperform COPY and are comparable with some older supervised systems, but underperform state-of-the-art supervised systems on WebNLG and E2E.Tables 3 and 4 report automatic metrics for WebNLG and E2E, respectively.Source: §7.1 · p. 7, Table 3 · p. 8, Table 4 · p. 8
  • The 2-STAGE system is generally on par with or better than 3-STAGE, while filtered-corpus models generally score better but produce more omissions.A separate aggregation stage offers explicit control over aggregation.Source: §7.1 · p. 7

7.2 Manual Error Analysis

  • Among 100 manually examined outputs, 1-STAGE tends to repeat facts and hallucinate; multi-stage systems largely eliminate these errors, but WebNLG outputs still merge unrelated facts and E2E outputs are generally consistent.On E2E, grammar errors and disfluencies are relatively minor and stem mainly from over-eager compression or template artifacts; Table 5 summarizes annotated error categories.Source: §7.2 · pp. 7–8, Table 5 · p. 8

7.3 Content Planning

  • The zero-shot ordering model beats random ordering and a Transformer-based approach without in-domain examples, while aggregation accuracy reaches 0.33 per example and 0.62 per sentence boundary versus random baselines of 0.23 and 0.50.The ordering results are summarized in Table 6; the aggregation evaluation reports room for improvement.Source: §7.3 · p. 9, Table 6 · p. 9
  • The paragraph-compression model follows the supplied content plan in 95% of sampled WebNLG cases and 100% of sampled E2E cases.The manual evaluation used 100 randomly selected examples with more than one triple from each dataset.Source: §7.3 · p. 9

7.4 Intrinsic Evaluation

  • On WikiFluent, paragraph compression scores highly when given ground-truth content plans, while ordering and aggregation performance is comparable to that on the D2T datasets.Table 7 reports results for full and filtered test sets.Source: §7.4 · p. 9, Table 7 · p. 9

8 Future Work

Future work targets template-free fact generation, task-specific pretraining, pipeline consistency checks, and more efficient modules. The authors identify improper fact merging as the approach’s main shortcoming and suggest controlling sentence-fusion semantics as a possible mitigation.

8 Future Work

  • Template-free fact generation could extend zero-shot systems to datasets with many predicates, while paragraph compression could serve as task-specific pretraining for more efficient finetuning with a small amount of clean data.The authors also propose pipeline consistency checks and more efficient architectures for individual modules.Source: §8 · p. 9
  • Improper fact merging remains the approach’s main shortcoming; explicitly controlling sentence-fusion semantics could mitigate it while retaining the zero-shot approach’s advantages.Source: §8 · p. 9

9 Conclusion

The paper presents a zero-shot D2T pipeline of PLMs trained on general-domain text operations, with three interpretable steps. It reports more semantically consistent outputs while cautioning that factual correctness is not guaranteed and neural D2T remains experimental.

9 Conclusion

  • The pipeline uses PLMs trained on general-domain lexical operations and three interpretable intermediate steps for zero-shot D2T generation.The authors suggest that training on large cross-domain corpora may support systems that perform well across domains.Source: §9 · p. 9
  • The authors report more semantically consistent outputs by avoiding noisy human-written references from D2T datasets.They also state that the approach helps reduce omissions and hallucinations stemming from reference noise, while conveying data semantics without implicit world knowledge.Source: §9 · p. 9, §10 · p. 10

10 Limitations and Broader Impact

  • The study focuses on RDF-triple descriptions, while neural D2T generation, especially zero-shot generation, remains limited to experimental settings.Source: §10 · p. 10
  • Because PLMs can reflect pretraining-corpus biases and output factual correctness cannot be guaranteed, the system should be used with caution.The paper notes possible reliance on spurious correlations when verbalizing attributes such as gender or occupation.Source: §10 · p. 10

A Data Statistics

The appendices describe template construction and its disfluencies, alongside additional E2E slot-error evaluation whose automated flags often did not match manual inspection.

B Templates

  • Templates were authored from input data without human references; extracting WebNLG templates from training examples was rejected because those examples were noisy and unavailable in zero-shot settings.Source: B Templates · p. 14
  • Using general templates can produce minor disfluencies, and the final pipeline step may fix some by rephrasing.The templates were mostly unambiguous, but some predicates required a more general wording.Source: B Templates · pp. 14–15

D Additional Results

  • Manual inspection found that most errors flagged by the E2E slot-error script were false, although some genuine hallucinations were also identified.One script pattern incorrectly flags an extra priceRange[high] slot in text mentioning price range and high customer rating.Source: D Additional Results · p. 15

E Example Outputs

The example-output appendix presents model behavior on WebNLG and E2E, with color coding for correctness and numbered facts used to reference ordering and aggregation.

E Example Outputs

  • Green marks completely correct outputs and red marks errors; numbered input facts serve as references in the Order and Aggregation rows.Source: E Example Outputs · p. 15
  • The WebNLG examples include incorrect outputs with hallucinations, improper fact merging, or pronoun-driven attribution errors.The captions distinguish these error patterns across Tables 11–13.Source: Table 11 · p. 16, Table 12 · p. 17, Table 13 · p. 17
  • The E2E examples contrast correct handling of many input values with redundant information added by 1-STAGE models.Source: Table 14 · p. 18, Table 15 · p. 19

Related papers

Loading 2203.16279v1…