PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Generative Models for Graph-to-Text Generation

Shuzhou Yuan, Michael Färber

arXiv:2307.14712v1cs.CLcs.AI

TL;DR

Because graph-to-text fine-tuning requires substantial resources and annotations, this paper evaluates GPT-3 and ChatGPT in a zero-shot setting on AGENDA and WebNLG. The models generate text, but fine-tuned systems outperform them, and the analysis finds relation errors and hallucinations.

  • Problem

    Zero-shot graph-to-text generation remains challenging, and evaluating generative models on graph data matters because training graph-to-text models requires substantial resources and annotations.

    Source: §1 · p. 1

  • Method

    The paper prompts GPT-3 and ChatGPT with linearized graph inputs and evaluates their generated text on AGENDA and WebNLG using multiple text metrics and error analysis.

    Source: §1 · pp. 1–2, §4 · p. 3, §6 · p. 6

  • Results

    Generative models fall short of fine-tuned models on both benchmarks, and their outputs often miss entity relationships or include unrelated information and hallucinations.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The authors identify refining generated text and reducing hallucinations as future priorities for graph-to-text generation.

    Source: §6 · p. 7

    Generated text and collected datasets may contain fake facts or offensive content; the authors do not endorse the generated text.

    Source: §7 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) have been widely employed for graph-to-text generation tasks. However, the process of finetuning LLMs requires significant training resources and annotation work. In this paper, we explore the capability of generative models to generate descriptive text from graph data in a zero-shot setting. Specifically, we evaluate GPT-3 and ChatGPT on two graph-to-text datasets and compare their performance with that of finetuned LLM models such as T5 and BART. Our results demonstrate that generative models are capable of generating fluent and coherent text, achieving BLEU scores of 10.57 and 11.08 for the AGENDA and WebNLG datasets, respectively. However, our error analysis reveals that generative models still struggle with understanding the semantic relations between entities, and they also tend to generate text with hallucinations or irrelevant information. As a part of error analysis, we utilize BERT to detect machine-generated text and achieve high macro-F1 scores. We have made the text generated by generative models publicly available.

1 Introduction

Zero-shot graph-to-text generation is understudied, despite the training resources and annotations required by existing approaches. This study evaluates GPT-3 and ChatGPT on two benchmarks and analyzes their outputs.

1 Introduction

  • Zero-shot graph-to-text generation remains challenging because LLM pretraining and fine-tuning use inconsistent input formats, while training graph-to-text models requires substantial resources and annotations.The paper identifies structured-data processing, particularly graph data, as understudied.Source: §1 · p. 1
  • The study prompts GPT-3 and ChatGPT to generate text from linearized graphs and evaluates them on the AGENDA and WebNLG test sets.The two datasets represent scholarly and general domains.Source: §1 · pp. 1–2
  • The authors compare generated text with references, find that generative models fall short of state-of-the-art approaches, analyze errors, train BERT to detect generated text, and release the outputs.The study presents its quantitative analysis as evidence about generative models' effectiveness in graph-to-text generation.Source: §1 · p. 2

2 Related Work

Prior graph-to-text work includes graph encoders, language models that use linearized graphs, and recurrent networks. This paper focuses on generative language models' ability to handle structured data.

Graph-to-text generation

  • Graph-to-text methods have also used GNN encoders and RNNs such as LSTM and GRU [Koncel-Kedziorski et al., 2019; Ribeiro et al., 2019; Song et al., 2018].The section groups prior approaches into GNN-based and LM-based types, while noting recurrent alternatives.Source: §2 · p. 2
  • GNN-based approaches encode graph structure, while LM-based approaches process token sequences after graphs are linearized.The section also identifies RNN approaches, including LSTM and GRU models.Source: §2 · p. 2

Generative Models

  • GPT-3, InstructGPT, and ChatGPT are generative language models; this work examines their capability to handle structured data.The section describes InstructGPT and ChatGPT as fine-tuned with supervised learning and then human-feedback reinforcement learning (Ouyang et al., 2022).Source: §2 · pp. 2–3

3 Dataset

The evaluation uses AGENDA and WebNLG test sets, which cover scholarly and general domains. Their inputs pair graphs with different forms of descriptive text.

3 Dataset

  • The experiments use AGENDA and WebNLG test sets because they are widely used and represent scholarly and general domains, respectively.The models require no further training in this evaluation.Source: §3 · p. 3
  • AGENDA pairs knowledge graphs automatically extracted by SciIE with scientific-paper abstracts, and the model input includes each paper's title, entities, and graph.Each instance also contains the abstract as its paired text.Source: §3 · p. 3
  • WebNLG maps DBpedia RDF-triple subgraphs to descriptions of one or a few sentences; the experiments use the 2017 challenge test data.Source: §3 · p. 3

4 Experiments

The experiments prompt GPT-3 and ChatGPT with linearized graphs and compare their outputs with fine-tuned T5 and BART results. Fine-tuned models outperform the generative models on both benchmarks, though the stronger generative model differs by dataset.

4 Experiments

  • The prompts request paper abstracts from title, entities, and graph for AGENDA, and text from graph for WebNLG, using linearized graph inputs.The inputs use GPT-3 text-davinci-003 and ChatGPT gpt-3.5-turbo-0301; each instance is a single request and the first response is used.Source: §4 · p. 3
  • The comparison uses Ribeiro et al. (2021a)'s fine-tuned T5 and BART results as baselines and evaluates generated text with BLEU, METEOR, RougeL, Chrf++, and BLEURT.BLEURT is used to assess semantic meaning and coherence beyond surface n-gram matching.Source: §4 · p. 3

4.1 Results

  • On AGENDA, ChatGPT scores 10.57 BLEU versus GPT-3's 8.34; on WebNLG, GPT-3 scores 20.36 versus ChatGPT's 11.08, while fine-tuned T5 and BART exceed 20 and 50, respectively.Source: Table 2 · p. 4, Table 3 · p. 4, §4.1 · p. 4
  • On WebNLG, GPT-3 leads ChatGPT by 9.28 BLEU, 3.06 METEOR, 9.77 RougeL, and 9.20 Chrf++; ChatGPT tends to hallucinate and generate longer text.Source: §4.1 · p. 4

5 Error Analysis

The error analysis finds that fluent generations can misrepresent graph relations and add irrelevant or hallucinated information. BERT also distinguishes generated text from human-written references with high scores.

5 Error Analysis

  • Generations can be fluent yet contradict graph facts: GPT-3 and ChatGPT reverse the accuracy–correlation-analysis relation, and ChatGPT ignores PART-OF and USED-FOR.These errors can produce inaccurate text.Source: §5 · p. 4, Table 4 · p. 5
  • The examples also show incorrect facts and hallucinations: GPT-3 treats Abner W. Sibal as a battle, while ChatGPT adds redundant internal knowledge and subjective judgments.The subjective judgment may cause unnecessary bias to potential users.Source: §5 · p. 5, Table 5 · p. 6
  • Fine-tuned BERT achieves high scores across AGENDA, WebNLG, and combined datasets, indicating that a classifier can distinguish generated from human-written text.The authors interpret this as generated text following similar patterns.Source: §5 · pp. 5–6

6 Conclusion

The paper evaluates zero-shot GPT-3 and ChatGPT for graph-to-text generation using linearized graphs and prompts. They do not surpass previously fine-tuned models, and the analysis identifies relation errors, hallucinations, and high BERT detection scores.

6 Conclusion

  • Zero-shot GPT-3 and ChatGPT fall short of previously trained and fine-tuned models on graph-to-text generation benchmarks.The study uses linearized graphs with prompts and evaluates the models on AGENDA and WebNLG.Source: §6 · pp. 6–7
  • The models struggle to capture entity relationships and often generate unrelated information or hallucinations.The authors suggest future work focus on refining generated text and reducing hallucinations.Source: §6 · p. 7
  • Fine-tuned BERT achieves high F1 scores when distinguishing machine-generated text from human-written text.Table 7 reports BERT detection results for GPT-3- and ChatGPT-generated text.Source: §6 · p. 7, Table 7 · p. 7

7 Ethical Consideration and Limitation

The authors flag potentially problematic content in both generated text and collected datasets. They also report that varying responses to identical requests undermine reproducibility and introduce randomness into evaluation scores.

  • Generative models may produce text containing fake facts or offensive content.Source: §7 · p. 7
  • The collected datasets may also contain incorrect or offensive statements.Source: §7 · p. 7
  • GPT-3 and ChatGPT often return different responses to the same request, making results hard to reproduce and evaluation scores random.Source: §7 · p. 7
Loading 2307.14712v1…