PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation

Zdeněk Kasner, Ondřej Dušek

arXiv:2401.10186v3cs.CL

TL;DR

D2T generation with LLMs remains underexplored, while standard benchmarks are saturated and training-data contamination further restricts experiments. The paper studies open models using newly collected structured records and finds fluent zero-shot text but frequent semantic errors.

  • Problem

    D2T generation with LLMs has limited evaluation beyond established benchmarks, which are saturated and vulnerable to training-data contamination.

    Source: §1 · p. 1

  • Method

    The authors introduce an approach using novel real-world structured records and reference-free evaluation, combining human annotations with a GPT-4-based metric.

    Source: §1 · pp. 1–2

  • Results

    Over 80% of open-LLM outputs contain at least one semantic error, despite fluent zero-shot generation from structured data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study provides word-level output annotations across five domains and recommendations for future directions in D2T generation.

    Source: §7 · p. 9

    Reference-free metrics rely on closed models and specific human annotation protocols, limiting replicability and making execution costly.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We analyze the behaviors of open large language models (LLMs) on the task of data-to-text (D2T) generation, i.e., generating coherent and relevant text from structured data. To avoid the issue of LLM training data contamination with standard benchmarks, we design Quintd - a tool for collecting novel structured data records from public APIs. We find that open LLMs (Llama 2, Mistral, and Zephyr) can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd. However, we show that the semantic accuracy of the outputs is a major issue: both according to human annotators and our reference-free metric based on GPT-4, more than 80% of the outputs of open LLMs contain at least one semantic error. We publicly release the code, data, and model outputs.

1 Introduction

D2T generation with LLMs remains underexplored, while standard benchmarks are saturated and training-data contamination further restricts experiments. The paper studies open LLMs on newly collected structured records and finds fluent zero-shot outputs alongside frequent semantic errors.

1 Introduction

  • D2T generation remains underexplored beyond a handful of established benchmarks, which are saturated and favor reference metrics that correlate poorly with human judgment.Training-data contamination of standard benchmarks further restricts the space for experiments.Source: §1 · p. 1
  • QUINTD collects novel structured records from public APIs across five D2T tasks, creating a 1,000-input ad-hoc benchmark in common formats for zero-shot testing.The tasks include weather forecasts, sports reports, product descriptions, chart captions, and entity descriptions.Source: §1 · pp. 1–2
  • The study evaluates three 7B open models, with GPT-3.5 as a closed-model baseline, using human annotations and a GPT-4-based reference-free metric for word-level semantic errors.Reference-free evaluation compares generated text against the input data rather than reference outputs.Source: §1 · p. 2
  • Over 80% of open-LLM outputs contain at least one semantic error, although the models generate fluent text from structured data in zero-shot settings.The finding is reported by both human annotators and a GPT-4-based metric.Source: §1 · p. 2
  • Long inputs create context-length and GPU-memory demands and make few-shot prompting unavailable, while preprocessing rules such as adding units and removing fields can improve outputs.The authors also recommend prefixing the model answer.Source: §1 · p. 2

2 Reference-Free D2T Generation

QUINTD collects ad-hoc test data from public APIs to support D2T evaluation without human references. QUINTD-1 comprises unlabeled records, and the task is to generate natural-language text faithful to each record.

2.1 Data Collection Tool

  • QUINTD uses public APIs to create ad-hoc test sets, addressing potential benchmark contamination and relying on plentiful structured data to keep collection lightweight and replicable.The records use common formats including JSON, CSV, and Markdown.Source: §2.1 · pp. 2–3

2.2 QUINTD-1 Dataset

  • QUINTD-1 has 500 development and 500 test examples, with 100 examples per domain per split, collected from November 2023 to January 2024 without reference outputs.Because the records are unlabeled, evaluation uses reference-free metrics; new benchmark versions can be generated with QUINTD.Source: §2.2 · p. 3

2.3 Task Definition

  • Given a structured record x from one of five domains and prompt P, the task is to generate natural-language output y faithful to x and the prompt instructions.Source: §2.3 · p. 3

3 Experiments

The experiments tested open LLMs on data-to-text generation with a shared prompt refined on development outputs. Preliminary observations highlighted fluent but semantically error-prone outputs, input constraints, and preprocessing choices used in final experiments.

3.1 Experimental Process

  • The authors used one prompt template across domains and models, iteratively refining the setup by inspecting development-set outputs.They aimed to limit preprocessing and prompt engineering because these could harm reproducibility and generalizability.Source: §3.1 · p. 3

3.2 Models

  • The open-model set comprised 7B instruction-tuned Llama 2, Mistral, and Zephyr models with 32k context, alongside GPT-3.5 for comparison.The open models fit on one NVIDIA A40 in 16-bit precision.Source: §3.2 · p. 3

3.3 Observations from Preliminary Experiments

  • Models sometimes surfaced irrelevant metadata and guessed ambiguous units, prompting field removal and explicit unit insertion; a fixed output prefix also made parsing easier.The authors retained understandable field names without adding descriptions.Source: §3.3 · p. 4
  • Long inputs required 32k-context models and downsampling, while context limits prevented robust few-shot prompting and half-shot examples introduced unrelated entities.Inputs in some domains exceeded 10–20k tokens; the authors downsampled owid and openweather data to under about 8k tokens.Source: §3.3 · p. 4
  • Most outputs were grammatically and stylistically fluent, yet contained semantic errors despite prompts emphasizing accuracy.This observation motivated word-level semantic-error evaluation.Source: §3.3 · p. 5

3.4 Final Experiments

  • Final test outputs used preprocessed inputs, Figure 2 prompts, and deterministic decoding capped at 512 tokens; GPT-3.5 decoding parameters were not fully accessible.Preprocessing stripped unnecessary fields, added units, and downsampled data to fit the context.Source: §3.4 · p. 5

4 Evaluation

The evaluation checks generated text against its input data for unsupported content, marking potential errors at the word level. It combines crowdworker annotations with a GPT-4-based metric and applies a four-category error taxonomy.

4 Evaluation

  • Evaluation compares generated text with input data and treats every output word as a potential source of an unsupported-content error.Errors are annotated at the word level.Source: §4 · p. 5
  • The two referenceless methods combine human judgments with a GPT-4 metric, balancing human-evaluation cost against concerns about relying only on a closed LLM.Human evaluation costs about ten times more; reporting both methods is intended to improve robustness and replicability.Source: §4 · p. 5
  • The taxonomy defines four high-level categories and avoids finer distinctions such as incorrect names versus incorrect numbers.Table 2 presents the categories with an example illustrating the error types.Source: §4 · p. 5, Table 2 · p. 6

4.1 Human-based Evaluation

  • Human evaluation used custom visualizations and 100 Prolific annotators, each labeling 20 examples across five domains.Annotators marked text spans with error categories; the median annotation time was 47 minutes.Source: §4.1 · p. 5

4.2 GPT-4-based Evaluation

  • The GPT-4 metric returns ordered JSON error spans with reasons, then aligns those spans to the output by sequential string matching.The authors report that requiring a free-form reason appeared in cursory observations to improve precision at almost no extra cost.Source: §4.2 · p. 6

5 Results and Discussion

Across five domains, open LLM outputs frequently contain semantic errors, and human and GPT-4 evaluations agree more strongly at broader aggregation levels than on individual words. The authors recommend prioritizing semantic accuracy while improving evaluation and testing practices.

5.1 How Accurate Are the Model Outputs?

  • Differences among open LLMs were not major; Zephyr ranked best across categories and metrics, while GPT-3.5 still made errors in 60-75% of examples.Mistral outputs were longer on average; GPT-3.5 and Llama 3 had lower correlations with human judgments in the reported scenario.Source: §5.1 · p. 6, §5 · p. 6
  • Open LLMs averaged more than two data-contradicting statements per output; NOT_CHECKABLE errors averaged over one per output (Ehum) and appeared in more than 25% of examples under both metrics.Error patterns varied by domain: Wikidata had 1.01 NOT_CHECKABLE versus 0.11 INCORRECT errors per output (Ehum), while Openweather outputs averaged over eight errors and more than 90% contained an error.Source: §5.1 · p. 6, Table 3 · p. 7
  • Depending on the model, 76-86% of examples contained errors according to Ehum, while 89-94% did according to Egpt.Table 4 reports the percentage of outputs containing at least one error.Source: §5.1 · p. 6, Table 4 · p. 7

5.2 Do Evaluation Methods Agree?

  • Ehum-Egpt correlation increased with aggregation (rword 0.26, rexample 0.52, rdomain 0.93), but the metrics marked the same words in fewer than 6% of cases.Human annotators showed a similar increase across levels (rword 0.36, rexample 0.53, rdomain 0.85); phrase-level error measurement is suggested as a possible way to improve agreement.Source: §5.2 · p. 7, Table 5 · p. 7

5.3 Recommendations for Future Work

  • The authors recommend focusing on semantic accuracy, developing model-based evaluation metrics, using efficient long-context models, and carefully checking API calls.They note that long contexts made few-shot experiments infeasible and that subtle API bugs could skew performance.Source: §5.3 · pp. 7–8
  • They also recommend continuous evaluation with human users, assessing real-world impact, and exploring multilingual D2T generation beyond English.The study itself evaluates English outputs.Source: §5.3 · p. 8

6 Related Work

Prior work addresses reference-based and reference-free text evaluation, as well as several established data-to-text tasks. Studies of LLMs in D2T have evaluated closed models on existing datasets, while this work examines open models.

6.1 Evaluation of Generated Texts

  • Text generation evaluation lacks a generally accepted solution, and human-written references are often noisy and expensive to collect (Celikyilmaz et al., 2020; Dušek et al., 2019, 2020).LLM-based reference-free evaluation has reported high correlations with human judgments, alongside concerns about reproducibility and model bias (Zhao et al., 2023; Kocmi and Federmann, 2023a; Wang et al., 2023b).Source: §6.1 · p. 8

6.2 D2T Generation Tasks

  • Earlier weather-generation work used templates, statistical methods, or tree-structured meaning representations; this study instead uses less-structured five-day outlooks (Belz, 2005, 2008; Angeli et al., 2010; Balakrishnan et al., 2019).Product-description work includes laptop and TV datasets and approaches ranging from recurrent models and templates to pretrained-language-model architectures (Wen et al., 2015, 2016; Wang et al., 2017; Shao et al., 2021; Koto et al., 2022).Source: §6.2 · p. 8
  • Prior sports-report datasets focus on basketball or baseball, whereas this study uses simpler ice-hockey summaries; chart-caption work has also moved toward large-scale datasets (Wiseman et al., 2017; Puduppully et al., 2019b; Obeid and Hoque, 2020; Kantharaj et al., 2022).Its chart task describes health-domain time series, while its entity task describes knowledge-graph entities from property lists, related to Wikipedia infotable generation.Source: §6.2 · pp. 8–9

6.3 D2T Generation with LLMs

  • Prior studies evaluated closed LLMs on WebNLG and AGENDA, finding them behind fine-tuned models on automatic metrics, while another study found GPT-4 could outperform state of the art on faithfulness.Source: §6.3 · p. 9

7 Conclusion

The paper presents an exploratory study of open LLMs for D2T generation, using ad-hoc test sets, common data formats, and reference-free evaluation. Across five domains, it combines GPT-4-based and human evaluation to provide word-level annotations and recommendations for future work.

7. Conclusion

  • The study explores open LLMs for D2T generation using ad-hoc test sets, common data formats, and reference-free evaluation.Source: §7 · p. 9
  • It evaluates model performance across five domains with a GPT-4-based metric and human evaluation, producing word-level annotations of model outputs.Source: §7 · p. 9
  • The authors provide recommendations for future directions in D2T generation based on their evaluation.Source: §7 · p. 9

Limitations

The study offers behavioral insights into open LLMs for data-to-text generation but does not compare them with other D2T approaches. Its scope and evaluation also have constraints tied to model size, reference-free metrics, and possible factual or semantic errors.

Limitations

  • The study does not compare open LLMs with other D2T approaches because its reference-free benchmark differs from prior methods largely finetuned on reference outputs.The authors frame behavioral insights into open LLMs as the study’s main goal despite this comparison boundary.Source: Limitations · p. 9
  • Reference-free metrics rely on closed models and specific annotation protocols, limiting replicability and raising costs; they can also err and correlate only moderately with one another.The authors expect open model-based metrics may address these shortcomings in the future.Source: Limitations · p. 9
  • The study uses only 7B-parameter open LLMs because of limited computational resources, and LLM outputs may contain factual or semantic errors.The authors state that hallucination causes are outside the paper’s scope and advise against relying on generated text alone for decisions.Source: Limitations · p. 9

Ethical Considerations

The paper describes its human-evaluation procedures and QUINTD data collection. Its data-source selection emphasized public availability, relevance to D2T, and common formats, while API limits were judged manageable at the collection scale.

Ethical Considerations

  • Annotators were paid the platform-recommended wage, preselected for English and residence in five countries, given detailed instructions, and shown data without sensitive or potentially offensive content.The study did not collect participant demographic data and received internal ethics committee approval.Source: Ethical Considerations · p. 9

A QUINTD Data

  • The QUINTD Data section describes the sources and procedure used to collect QUINTD-1, and points to scripts for replicating collection.Table 6 summarizes output types for each domain.Source: A QUINTD Data · p. 14

A.1 Selection of Data Sources

  • Data sources were required to be public, represent a common D2T task, and use a common or readily transformable format; API access limits did not severely constrain collection at this scale.All selected sources were accessible through APIs.Source: §A.1 · p. 14

A.2 Data Collection

QUINTD’s data collection draws on five domains and formats, with sources selected to support distinct D2T tasks. The appendix describes the source-specific sampling and formatting procedures, alongside human and GPT-4 evaluation implementation details.

A.2.1 OpenWeather

  • OpenWeather’s forecast5 API supplied five-day forecasts at three-hour resolution; the free tier allowed 1,000 daily calls, but its temporal restrictions limited exact dataset replication.The 1,000 weather records were downloaded on Jan 3, 2024, using randomly sampled cities for each split.Source: §A.2.1 · p. 15

A.2.2 GSMArena

  • GSMArena specifications were obtained through an unofficial API, with a balanced sample drawn by downloading 10 products per available brand before selecting 100 products per split.The API’s per-IP request limits may delay larger downloads.Source: §A.2.2 · p. 15

A.2.3 RapidAPI Ice Hockey

  • IceHockeyAPI match records covered games on Nov 27, 2023, for development and Nov 29 for test; 100 games were randomly sampled for each split.The endpoint returned all games for a day in one request, so its 50-request daily limit did not constrain collection.Source: §A.2.3 · p. 15

A.2.4 OurWorldInData

  • OurWorldInData collection was restricted to health time series, with 100 examples per split sampled from countries having non-empty entries and formatted as two-column CSV.The chosen series support chart descriptions of value changes over time; inputs included title, description, and unit as comments.Source: §A.2.4 · p. 15

A.2.5 Wikidata

  • Wikidata subgraphs came from companies, countries, films, and humans, with 2–10 properties per entity and 100 randomly sampled subgraphs per split.Inputs used Markdown snippets with the entity as a title and each key-value pair as a bullet.Source: §A.2.5 · pp. 15–16

B Human Evaluation / C GPT-4 Evaluation / C.1 Aligning the Errors

  • Human review used domain-specific visualizations; GPT-4 scoring forced JSON at temperature 0, but string-match alignment failed in 137/6,927 cases and a revised prompt example was not rerun.The study used interactive graphs for OpenWeather and OurWorldInData and manually created tables for other domains; alignment matched GPT-4’s quoted spans.Source: B Human Evaluation · p. 16, C GPT-4 Evaluation · p. 16, §C.1 · p. 16

D Experiments with Open LLMs as Evaluators

Among the tested evaluators, GPT-4 had the strongest correlation with human annotations, followed by Llama-3-70B and GPT-3.5, so the authors selected GPT-4. The appendix also provides examples of inputs and outputs across the five domains.

D Experiments with Open LLMs as Evaluators

  • GPT-4 had the strongest Pearson correlation with human annotations, followed by Llama-3-70B and GPT-3.5; the authors chose GPT-4 because the gap between models was substantial.The models used the same prompts, temperature 0, and force-decoded JSON outputs.Source: D Experiments with Open LLMs as… · p. 16, Table 7 · p. 16

E Examples

  • The appendix shows input-output examples for all five domains, with interactive graphs for OpenWeather and OurWorldInData available through the web interface.The examples pair figures and tables across OpenWeather, GSMArena, ice hockey, OurWorldInData, and Wikidata.Source: E Examples · pp. 16–17

F Full Results

The appendix identifies per-domain error measures and illustrates the annotation criteria with examples of generated text paired with input data.

How should I decide on the errors?

  • The annotation criteria include facts that contradict the data, cannot be checked from it, or are misleading in context.Source: F Full Results · p. 19

What is the purpose of the research?

F Full Results

  • Tables 13 and 14 report, respectively, average errors per output and the percentage of outputs with at least one error, separately by domain.Both captions say lower values are better; the passages do not provide the tables’ cell values.Source: F Full Results · p. 17, Table 13 · p. 28, Table 14 · p. 28

Related papers

Loading 2401.10186v3…