Source-linked AI summary
An Empirical Study of Recent Large Language Models for Knowledge Graph to Text Generation on Standard Benchmarks
Priyanka, Renaud Delbru, Souhail Bakkali, Mickaël Coustaty, Antoine Doucet
TL;DR
KG-to-text research needs to assess whether recent LLMs approach fine-tuned models and how generated text aligns with its source graph. This paper benchmarks recent models using reference-based metrics and graph-grounded LLM judges validated against human annotations. Prompted models come close on reference-based metrics and perform better on graph-grounded evaluation, which exposes differences that reference metrics may not capture.
Problem
The paper asks how recent prompted LLMs compare with fine-tuned models on KG-to-text benchmarks and whether existing metrics capture graph faithfulness and coverage.
Source: §1 · pp. 1–2Method
The study benchmarks recent model families against fine-tuned T5-large using reference-based metrics and graph-grounded LLM judges, with judge evaluation validated against human annotations.
Source: §1 · pp. 1–2Results
Prompted decoder-only models stand close to, and sometimes slightly ahead of, T5-large on reference-based evaluation while clearly outperforming it on source-grounded evaluation.
Source: §7 · p. 8Takeaways & Limitations
LLM-based and reference-based evaluations capture different aspects of generation quality, and graph-grounded evaluation distinguishes the faithfulness errors of prompted and fine-tuned systems.
Source: §7 · p. 8The benchmark datasets are not independent by construction: DART incorporates WebNLG17 data, and WebNLG20 extends WebNLG17’s underlying triple set.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Knowledge graph-to-text (KG-to-text) generation verbalizes a set of knowledge graph triples into fluent natural language. While fine-tuned encoder-decoder models have long been the standard approach, recent LLMs are becoming increasingly competitive. We benchmark recent LLMs against fine-tuned encoder-decoder models, using reference-based metrics and graph-grounded reference-free LLM-based evaluation validated against human annotations. We observed that recent LLMs substantially narrow, and in several settings close, the gap with fine-tuned models on reference-based metrics, while performing better on graph-grounded metrics. LLM-based evaluation scores show limited correlation with reference-based metrics, suggesting that the two approaches capture different aspects of generation quality. Graph-grounded evaluation reveals that prompted models produce fewer contradictions and triple omissions than fine-tuned models, and these errors increase more rapidly with graph size for fine-tuned models, highlighting differences between the two approaches that reference-based metrics may not fully capture.
1 Introduction
The paper evaluates recent LLMs against fine-tuned T5-large for KG-to-text generation and examines how reference-based and LLM-judge metrics compare. It also investigates whether prompted and fine-tuned systems produce different faithfulness errors.
Research scope
- The study evaluates five recent model families against fine-tuned T5-large on WebNLG17, WebNLG20, and DART using eight reference-based metrics.It also studies source-grounded or reference-free LLM judges and their reliability for judging generated text.Source: §1 · pp. 1–2
Research questions
- The paper asks whether recent prompted and LoRA-adapted models approach fully fine-tuned T5-large across metrics and datasets.Its other questions concern metric agreement, LLM-judge validity against human annotations, and whether prompted and fine-tuned systems make the same errors.Source: §1 · p. 2
Reported contributions
- The authors report that recent models narrow the gap with fine-tuned models, with parity on neural metrics but a remaining fine-tuning lead on BLEU, BARTScore, and PARENT.They also report weak agreement between reference-based metrics and a source-grounded judge, and different faithfulness error types for prompted and fine-tuned systems.Source: §1 · p. 2
2 Related work
KG-to-text work has moved from recurrent and graph neural architectures toward pretrained language models, while evaluation has expanded from references and human annotation to LLM judges. The study selects datasets intended to have low graph-text alignment noise.
Generation approaches
- Early KG-to-text systems used RNNs, including LSTM and GRU, followed by GNN-based models intended to capture graph structure and relations.More recent approaches increasingly use pretrained language models with linearized graph inputs.Source: §2 · p. 2
Dataset choice
- WebNLG17, WebNLG20, and DART were selected because their training data have negligible graph-text alignment noise.The cited alternatives include datasets with hallucinated entities or relations, unused triples, or unguessable text.Source: §2 · p. 2
Recent LLM studies
- Prior work found zero-shot GPT-3 significantly behind fine-tuned sequence-to-sequence systems, while detailed prompts improved GPT-4o results on WebNLG20.This study extends evaluation to a broader set of recent LLM architectures.Source: §2 · p. 2
Evaluation approaches
- Evaluation has used reference-based metrics and human annotation, with LLM judges explored as a scalable automatic alternative.Check-Eval and BinEval decompose judgments into binary questions, while another approach localizes mismatches with a fine-tuned verifier.Source: §2 · pp. 2–3
3 Datasets
The study evaluates KG-to-text generation on WebNLG17, WebNLG20, and DART, using the full test subsets or datasets described in the paper. The benchmarks cover RDF verbalization, an expanded WebNLG category set, and open-domain graphs from diverse sources.
WebNLG17
- WebNLG17 is an RDF-triple verbalization benchmark with graphs of 1 to 7 triples, and experiments use its full test subset.It was released as a common benchmark for comparing microplanners for RDF description.Source: §3 · p. 3
WebNLG20
- WebNLG20 adds a company category to categories present in WebNLG17, and experiments use its full test dataset.The dataset has been revised several times since the first challenge.Source: §3 · p. 3
DART
- DART is an open-domain graph-to-text benchmark built from human-annotated tables, converted questions, and RDF-to-text datasets.Its sources include WikiSQL, WikiTableQuestions, WebNLG 2017, and E2E.Source: §3 · p. 3
4 Model Architectures
The evaluated checkpoints span encoder-decoder and decoder-only designs, dense and sparse models, and open and closed-source systems. The model set ranges from 270M parameters to a 550B-parameter sparse model.
Architecture range
- The selected checkpoints range from 270M-parameter encoder-decoder models to a 550B-parameter sparse model.The architecture set includes encoder-decoder and decoder-only models, with dense and sparse variants.Source: §4 · p. 3
Architecture families
- T5 and T5-Gemma2 are encoder-decoder models, while the other listed models are decoder-only; Granite-4.2 is dense and Gemma-4 and Nemotron-3 include sparse variants.T5-Gemma2 adapts a pretrained decoder-only Gemma 3 model into an encoder-decoder, with tied embeddings and a merged attention module. GPT-5 serves as the closed-source reference, with nano, mini, and full variants.Source: §4 · p. 3
5 Experiments
The experiments fine-tune T5 variants and T5-Gemma2, evaluate prompted decoder-only models and GPT-5, and compare outputs using reference-based, graph-alignment, and LLM-judge metrics.
5.1 Fine-tuned models
- T5 variants were fine-tuned independently on WebNLG17, WebNLG20, and DART, while T5-Gemma2 used LoRA for the same task.T5 experiments used early stopping and beam search; T5-Gemma2's LoRA settings were selected through a hyperparameter sweep over ranks and learning rates.Source: §5.1 · pp. 3–4
5.2 Prompted models
- Decoder-only models and GPT-5 used few-shot prompts with a shared prompt format and graph linearization consistent with the T5 experiments.Few-shot examples were retrieved using graph-embedding similarity or graph triple-size subsets, with identical test instances removed from training data.Source: §5.2 · p. 4
5.3 Evaluation Protocol
- Evaluation covered lexical-overlap, semantic, graph-alignment, and reference-free LLM-judge metrics, including BinEval and GPTScore.BinEval assessed graph faithfulness, triple coverage, and fluency using binary checks; GPTScore evaluated output likelihood conditioned on the source graph and task instruction.Source: §5.3 · pp. 4–5
6 Results
Across three benchmarks, fine-tuned T5-large remained strongest on most reference-based metrics, but prompted models led on some datasets and metrics. LLM judges favored prompted systems, while human validation and metric correlations showed that graph-grounded and reference-based evaluations capture different aspects of output quality.
6.1 Standing of recent models on reference based evaluation (RQ1)
- For few-shot prompting, performance rose through three shots and declined thereafter, so the reported prompted-model results use three-shot settings.Cosine-similarity retrieval was used because alternatives showed no meaningful advantage.Source: §6.1 · pp. 5–6
- On WebNLG17, T5-large led on almost all metrics; on WebNLG20, Gemma-4-31B-it led on all metrics; and on DART, T5-large led except on BLEURT.Nemotron-3-Ultra matched T5-large on WebNLG17 METEOR and BLEURT, led WebNLG20 BARTScore, and was among the DART BLEURT leaders.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 6
6.2 LLMs-as-judge results (RQ3)
- Both LLM judges ranked prompted models above fine-tuned models, unlike reference-based metrics, which ranked T5 best in most scenarios.Fine-tuned systems had faithfulness failures on 12%–23% of outputs versus 2%–10% for prompted systems; fine-tuned failures more often corrupted whole outputs.Source: §6.2 · p. 6, Table 4 · p. 7, Table 5 · p. 7
- As graph size increased from one or two triples to four or more, faithfulness failures and missing triples rose more for fine-tuned models.Under DeepSeek, T5-large faithfulness failures rose from 8% to 26%, and missing triples from 3% to 27%.Source: §6.2 · p. 7
6.3 Validation against human annotations (RQ3, RQ4)
- Human validation found that judges detected hard contradictions and entity corruptions in T5-large outputs, but flagged only one of twelve unsupported elaborations.For triple omissions, they recovered most T5-large omissions and the single Nemotron-3-Ultra omission.Source: Table 6 · p. 7, §6.3 · p. 8
6.4 Agreement between metrics (RQ2)
- Across all six model outputs on WebNLG17, reference-based NLG metrics correlated strongly with one another (0.62–0.92), while their correlations with LLM faithfulness judges were lower (0.06–0.36).The two judges correlated best on fine-tuned models: 0.74 on T5-Gemma2 and 0.67 on T5-large.Source: §6.4 · p. 8
7 Conclusion
Prompted decoder-only models approached or sometimes surpassed T5-large on reference-based evaluation and outperformed it on source-grounded evaluation, while the judges exposed limitations of reference-based metrics and their own detection capabilities.
7 Conclusion
- Prompted decoder-only models came close to T5-large on reference-based evaluation, sometimes led on WebNLG20, and clearly outperformed it on source-grounded evaluation.The LoRA-tuned encoder-decoder performed like fully fine-tuned T5 on both evaluation approaches.Source: §7 · p. 8
- BinEval scores did not correlate strongly with reference-based metric families, although those families correlated well with one another across model outputs.The judges closely reproduced human judgments on contradictions and entity corruption but often missed unsupported elaborations.Source: §7 · p. 8
- Prompted models contained fewer contradictions and triple omissions than the fine-tuned baseline, while fine-tuned models became more prone to graph-grounded errors as graph size increased.The conclusion presents these as findings supported by BinEval evaluations.Source: §7 · p. 8
- Existing benchmarks use graphs of at most seven triples, and report-scale evaluation needs larger aligned datasets and protocols that detect ungrounded elaboration.The paper states that no current automatic evaluation reliably detects ungrounded elaboration.Source: §7 · p. 8
Limitations
The study’s findings are bounded by evaluation and sampling choices, a limited fine-tuning comparison, and overlap among the benchmark datasets.
Limitations
- Decoder-only models were evaluated with one generation per model, and the fine-tuning comparison covered T5-large and one LoRA-tuned model.The experiments did not extend fine-tuning to a decoder-only architecture.Source: Limitations · p. 9
- Human evaluation covered 110 of 1,862 WebNLG17 samples and only Nemotron-3-Ultra Faithfulness and Coverage, leaving Fluency unannotated.A single annotator rated outputs, with a second author adjudicating disputed verdicts while seeing the LLM judges’ explanations; agreement estimates have wide confidence intervals.Source: Limitations · p. 9
- WebNLG17, WebNLG20, and DART overlap by construction, so the study cannot rule out judge-LLM dataset memorization.DART incorporates WebNLG17 data, while WebNLG20 extends WebNLG17’s underlying triple set.Source: Limitations · p. 9
- GPTScore with DeepSeek-V4.1-Flash remained inconclusive, and the study did not analyze why.T5 fine-tuning scores also differed from some published values because of evaluation-protocol differences.Source: Limitations · p. 9
A Error Analysis of simpler prompt setting
Preliminary GPT-5 prompting produced structural-marker, semantic-ordering, and extra-comment errors; explicit instructions addressing these failures eliminated them in revised outputs.
A Error Analysis of simpler prompt setting
- The initial prompt left GPT-5-mini’s output quality highly dependent on the prompt, and the model sometimes treated <H>, <R>, and <T> as literal entity identifiers.It referred to the head, relation, and tail entities as “Entity H,” “Entity R,” and “Entity T.”Source: A Error Analysis of simpler prompt… · p. 11
- GPT-5 models also produced semantically nonsensical text that followed graph entity order and sometimes added explanatory comments or intermediate reasoning.The paper reports these as recurring preliminary issues.Source: A Error Analysis of simpler prompt… · p. 11
- Explicit instructions addressing the recurring failures in the detailed prompt eliminated those errors from generated outputs.The paper notes consistency with the error analysis reported by Yuan and Färber (2023).Source: A Error Analysis of simpler prompt… · pp. 11–12
B Bineval and GPTScore details
The paper evaluates generated graph descriptions with BinEval for faithfulness, triple coverage, and fluency, and uses GPTScore to measure likelihood given the graph and task instruction.
B Bineval and GPTScore details
- BinEval judges six models on three datasets for semantic faithfulness, graph-triple coverage, and fluency.Its faithfulness prompts present the graph as linearized triples alongside the candidate text.Source: B Bineval and GPTScore details · p. 12
- For each question, BinEval returns a JSON object with a boolean “satisfied” verdict and a brief “explanation.”Source: B Bineval and GPTScore details · p. 12
- For DART, the judge prompt explains that [TABLECONTEXT] denotes the record and that a [TITLE] relation’s tail gives its title or subject.The faithfulness prompt also includes three questions and violation examples not taken from the test sets.Source: B Bineval and GPTScore details · p. 12
- Coverage is the ratio of graph triples judged described in the text, while GPTScore evaluates the text’s likelihood given the graph and task instruction.Source: B Bineval and GPTScore details · p. 12