PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data

Yifei Song, Kun Efimov-Zhang, Claire Gardent

arXiv:2608.23391v1cs.CLcs.AI

TL;DR

Cross-domain data-to-text generation without target-domain training text or test references must contend with diverse schemas, formats, and long inputs. The paper applies data-driven knowledge distillation with structure-preserving augmentation, finding that 1.7B-parameter models achieve lower error rates than same-size zero-shot and WebNLG-finetuned baselines across five domains.

  • Problem

    Cross-domain data-to-text generation without target-domain training or reference text must handle substantial schema, format, and input-length heterogeneity.

    Source: §1 · pp. 1–2

  • Method

    The paper distills teacher-generated text for target-domain inputs into student models and augments those inputs through structural subsampling and perturbation.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    1.7B-parameter DDKD models had lower error rates than same-size zero-shot and WebNLG-finetuned baselines across all five target domains.

    Source: §1 · p. 2, §6.1 · p. 7

  • Takeaways & Limitations

    Cross-domain data-to-text capabilities can transfer to compact models when large LLMs serve as intermediate generators rather than deployment targets.

    Source: §7 · p. 9

    Because evaluation focuses primarily on Qwen3 and Gemma3 small-scale models, transferability of the gains to significantly larger architectures remains unestablished.

    Source: §8 · p. 9

Abstract

from arXiv · show

Structured data exists in many forms (tables, knowledge graphs, charts, and time series), and converting it into text may involve different generation tasks. However, most prior work on data-to-text (D2T) generation has focused on specific tasks and datasets, relying either on task-specific training data or on the zero-shot capabilities of large language models. We study cross-domain D2T generation in a setting where neither in-domain training text nor test references are available, and where domains, generation goals, and input structures vary substantially. We compare data-driven knowledge distillation (DDKD) against zero-shot inference and fine-tuning on out-of-domain D2T data, and introduce structure-preserving augmentation via structural subsampling and perturbation. Experiments on five benchmarks show that, at constant model size (1.7B parameters), DDKD consistently outperforms both fine-tuning and zero-shot inference. Moreover, the resulting small models outperform a much larger finetuned model on two of the five domains, achieving comparable performance on the remaining three. We further construct QUINTD-5, a fivefold extension of QUINTD-1, and show that simply scaling real target-domain inputs yields only modest gains, whereas our augmentation strategy remains more effective and more cost-efficient for cross-domain distillation.

1 Introduction

The paper studies cross-domain data-to-text generation without target-domain training text or test references, comparing zero-shot, out-of-domain fine-tuning, and data-driven knowledge distillation across heterogeneous data structures and domains.

1 Introduction

  • Across both model families, DDKD yields lower error rates than same-size zero-shot and WebNLG-finetuned baselines across all five target domains.Figure 1 reports per-domain error rates averaged over two independent LLM judges.Source: Figure 1 · p. 1, §1 · p. 2
  • The setting combines schema and format heterogeneity, unavailable target-domain training and reference data, long inputs that constrain prompting, and limited zero-shot generalisation.Inputs vary in fields, nesting, semantics, and JSON, CSV, or Markdown formats.Source: §1 · pp. 1–2
  • The study compares zero-shot prompting, out-of-domain fine-tuning, and distillation from zero-shot or D2T-finetuned teachers, while testing augmentation and content-coverage trade-offs.DDKD students train on synthetic target-domain examples; the fine-tuned teacher receives 40K WebNLG knowledge graph–text pairs.Source: §1 · p. 2
  • The paper introduces structural subsampling and perturbation, and constructs QUINTD-5 to show augmentation is more effective and cost-efficient than scaling real target-domain inputs.QUINTD-5 is a fivefold extension of QUINTD-1.Source: §1 · p. 2
  • The evaluation combines two independent LLM judges, human validation, agreement analysis, and a content-coverage check indicating DDKD improves faithfulness without conservative under-generation.The contribution describes a reference-free evaluation using an error taxonomy and LLM-as-a-Judge assessment.Source: §1 · p. 2

2 Related Work

Prior D2T work often targets a single or similar input structures, while existing distillation approaches are largely task-specific. This work combines out-of-domain training, synthetic data generation, and structured perturbation to support cross-domain D2T without target-domain references.

Data-to-Text Generation

  • Most D2T research targets one or similar input structures, with only a few approaches addressing multiple structures.Prior exceptions use decomposition and multistage generation or a unified framework for tables, graphs, and meaning representations.Source: §2 · pp. 2–3
  • Xiang et al. (2022) and Li et al. (2023a) address broader structures, but both rely on existing training data; Kasner and Dusek (2024) study LLMs across five domains and tasks.This work builds on the latter study by investigating modeling strategies for its datasets.Source: §2 · pp. 2–3

Knowledge Distillation

  • Distillation has been applied across NLP and NLG, while recent LLM work explores rationale supervision, on-policy learning, and domain-specific transfer.Prior data-to-text distillation includes reasoning-focused methods for scientific table-to-text, logical table-to-text, and graph-to-text.Source: §2 · p. 3
  • Unlike prior data-to-text distillation, this work targets cross-domain generation without target-domain references, combining out-of-domain training, synthetic data, and structured perturbation.Prior approaches described here focus on scientific table-to-text, logical table-to-text, or graph-to-text.Source: §2 · p. 3

3 Cross-Domain D2T Generation

The section frames cross-domain data-to-text generation as producing fluent, fact-faithful English from structured inputs under a prompt. It describes a reference-free evaluation across five domains using synthetic target-domain supervision and compact-student distillation.

Input Data

  • QUINTD-1 provides 100 development and 100 test inputs per domain without in-domain references; development inputs seed synthetic supervision, while test inputs are reserved for evaluation.The five domains vary substantially in generation objectives, input structures, and output formats.Source: §3 · pp. 3–4

Models

  • The study compares zero-shot prompting, WebNLG fine-tuning, and DDKD using either zero-shot or WebNLG-fine-tuned teachers; open-weight models are used for reproducible fine-tuning and distillation.The zero-shot comparison also includes GPT-4.1, while the open-weight models are Qwen3-32B and Gemma3-27B-IT.Source: §3 · p. 4

LLM Prompting (ZS)

  • More than 80% of texts from the previously tested open LLMs contained at least one semantic error, prompting comparison with newer models and GPT-4.1.GPT-4.1 is excluded from fine-tuning and distillation because it does not support reproducibility (Kasner and Dusek, 2024).Source: §3 · p. 4

Fine-Tuning on WebNLG data

  • WebNLG is selected for source initialization because its human-written, fact-aligned references span diverse DBpedia categories, and an ablation finds the best overall faithfulness–coverage trade-off.E2E yields competitive error counts on some simpler domains but substantially lower coverage; KELM-Q1 performs worse on both metrics across most domains (Novikova et al., 2017; Song and Gardent, 2025).Source: §3 · p. 4

DDKD from ZS and WebNLG-SFT Teachers

  • DDKD trains a compact student on teacher-generated input–text pairs, expanded with augmentation and perturbation, while keeping the large teacher’s role to offline conditional generation.The student uses maximum-likelihood training on synthetic texts; the setup is intended to make training and deployment more efficient than direct fine-tuning of large models.Source: §3 · pp. 4–5

4 Training Data

Training uses WebNLG for out-of-domain fine-tuning and QUINTD-1 development inputs for synthetic target-domain supervision. The section describes structure-preserving augmentation and a fivefold real-data scaling control.

4.1 WebNLG Data.

  • WebNLG provides 39,890 DBpedia knowledge-graph/text pairs across 16 categories for fine-tuning, with each graph deterministically linearized as JSON, Markdown, and CSV.Table 2 illustrates the three representations for the same underlying graph.Source: §4.1 · p. 5, §4 · p. 5, Table 2 · p. 6

4.2 Target-domain Synthetic Data Generation

  • Synthetic DDKD data starts from 100 QUINTD-1 development instances per domain, verbalized by a WebNLG-finetuned teacher, and adds examples from non-empty subsets of each input’s attributes.The additional QUINTD-1 data is collected through the QUINTD APIs; each selected subset is verbalized to create a corresponding synthetic text.Source: §4.2 · p. 5, §4 · p. 5
  • Structural perturbation modifies selected subsampled instances, with a fixed perturbation ratio of 20%; mixing and shuffling clean and perturbed examples exposes the student to varied inputs.The strategy aims to reduce overfitting and improve robustness to schema variation and partial structural noise.Source: §4.2 · p. 5

4.3 QUINTD-5: A Real-Data Scaling Control

  • QUINTD-5 expands QUINTD-1 from 100 to 500 real structured inputs per domain using the same collection pipeline, public APIs, and preprocessing.It supports comparing gains from more real target-domain inputs with gains from augmented supervision generated from a smaller seed set.Source: §4.3 · pp. 5–6

5 Experiments

The experiments compare Qwen3 and Gemma3, evaluate faithfulness with LLM judges because reference texts are unavailable, and use human evaluation to assess system outputs and the judges’ reliability.

5.1 Model Families and Ablation Studies

  • Qwen3 receives the full augmentation ablation, while Gemma3 tests transfer using Qwen3’s best configurations; experiments also include QUINTD-5 scaling and a content-coverage check.This design evaluates generality across two model families while keeping the experimental budget manageable.Source: §5.1 · p. 6

5.2 Automatic Evaluation

  • Because QUINTD-1 has no reference texts, faithfulness is evaluated by comparing outputs with structured inputs and averaging judge-identified errors across four categories.The categories are Incorrect Fact, Not Checkable, Misleading, and Other; Kasner and Dusek (2024) is the cited prior work for this protocol.Source: §5.2 · pp. 6–7
  • GPT-5.1 (Singh et al., 2025) is the primary judge, while Gemini-2.5-Pro independently evaluates systems and is compared with GPT-5.1 for agreement.A separate content-coverage check uses the same two judges on the subset used for human evaluation.Source: §5.2 · pp. 6–7

5.3 Human Evaluation

  • Human evaluation covers 240 outputs from 60 stratified inputs—12 per domain—across a small zero-shot model, source-domain SFT, the best distilled model, and the best large teacher.Annotators label error types and severity at the example level and mark error spans.Source: §5.3 · p. 7

6 Results and Discussion

At 1.7B parameters, data-driven knowledge distillation (DDKD) outperforms fine-tuning and zero-shot prompting, with distilled models surpassing much larger models on four of five domains. The results also show that structured augmentation improves distillation, while exposing the teacher to more raw inputs alone yields only modest gains.

6.1 Automatic Evaluation Results

  • NormAvg is 0.42 vs. 0.27 under the ZS teacher and 0.20 vs. 0.10 under the WebNLG-SFT teacher, with 500-real remaining behind the best augmented models.The authors suggest augmentation helps because structural subsampling creates more supervision from a small seed set and simplifies complex inputs while preserving their structure.Source: §6.1 · pp. 7–8
  • At 1.7B parameters, distilled models consistently outperform fine-tuning and zero-shot prompting, and outperform Qwen3-32B and GPT-4.1 on four of five domains.The four domains are Wikidata, Ice Hockey, GSM Arena, and OWID; the models use structured inputs and teacher-generated synthetic outputs, without human-written or task-specific target-domain references.Source: §6.1 · p. 7, Table 4 · p. 8

The structured data augmentation variants systematically outperform base configuration,

  • Perturbation-only or mixed subsampling/perturbation improves performance on four of five datasets, but augmentation is less effective for weather forecasting, which requires aggregation and summarization.The authors attribute the limitation to teacher models already abstracting away fine-grained attribute variations in this task.Source: The structured data augmentation… · p. 8
  • Weather has the worst scores across models; domain rankings also vary by training approach, with WebNLG fine-tuning benefiting Wikidata and zero-shot prompting outperforming it on GSM Arena.The authors suggest WebNLG supervision reinforces graph-to-text verbalization for Wikidata, while source-domain fine-tuning may bias the GSM Arena teacher toward overly structured generation.Source: The structured data augmentation… · p. 8
  • Not Checkable errors are 0.05 for the best DDKD model, versus 0.53 for zero-shot and 0.70 for fine-tuning; large models are more faithful to inputs, with Incorrect errors of 0.14, 0.50, and 0.41 versus 1.13 for DDKD.DDKD also achieves higher coverage than same-size zero-shot and source-domain fine-tuned baselines, remains close to the selected best teacher, and has high inter-judge agreement (r > 0.95).Source: The structured data augmentation… · pp. 8–9

6.2 Human Evaluation Results

  • Human annotator agreement is high for Incorrect Fact and Other (α = 0.810 and 0.871), and GPT-5.1 rankings correlate strongly with human system rankings (Pearson r > 0.90).Automatic and human evaluations show broadly consistent trends: distillation ranks first overall, OpenWeather scores are lowest, and small distilled models outperform the large teacher on several domains.Source: §6.2 · p. 9

7 Conclusion

The paper studies data-to-text generation across multiple input structures and generation objectives. Its findings indicate that compact models can benefit from large LLMs as intermediate generators, while augmentation and data-driven knowledge distillation help address missing training data.

7 Conclusion

  • Cross-domain generalisation can be transferred to compact models by using large LLMs as intermediate generators rather than final deployment targets.This frames large LLMs as a means of producing the transfer, not the deployed model.Source: §7 · p. 9
  • Data augmentation and data-driven knowledge distillation can help mitigate the absence of training data.Source: §7 · p. 9
  • These approaches enable the study of new generation tasks based on existing real-world data.Source: §7 · p. 9

8 Limitations · A Implementation and Training Details · A.1 Training Hyperparameters

The paper’s limitations concern model and evaluator scope, heuristic choices in structural augmentation, and the cost and domain sensitivity of LLM judging. Training uses long contexts, with distinct settings for source-domain and target-domain fine-tuning.

8 Limitations

  • Gains are established mainly for 1B–2B Qwen3 and Gemma3 models, so transfer to significantly larger architectures remains unestablished.The study prioritizes these open-weight families for their long-context support (>10k tokens) in the small-model regime, where long-sequence fine-tuning, inference, and data generation can be computationally prohibitive.Source: §8 · p. 9
  • GPT-4.1 is used only for comparison and GPT-5.1 only for evaluation, not training-data generation, supervision, or distillation; open evaluators remain future work.The reference-free setting motivates using frontier models for automated assessments, while the training and data-generation pipeline remains auditable and reproducible.Source: §8 · p. 9
  • Structural augmentation relies on heuristic atomic units and exchangeability constraints that may not suit every domain, though generic lightweight instantiations remain effective.These choices require minimal assumptions about input structure and introduce inductive biases.Source: §8 · pp. 9–10
  • LLM judging can be domain-sensitive and costly, but GPT-5.1 and Gemini-2.5-Pro judgments corroborate human evaluation’s consistent trends.The paper anticipates that open-model metrics could lower cost and improve accessibility.Source: §8 · p. 10

A.1 Training Hyperparameters

  • WebNLG source fine-tuning uses a 1 × 10^-4 learning rate, batch size 8, up to 10 epochs, and a 13,000-token maximum sequence length.Runs use MS-SWIFT, LoRA, early stopping, and—unless specified otherwise—a fixed seed of 42; LoRA rank is 8 and α = 32. Long contexts match downstream inference conditions.Source: §A.1 · p. 13
  • Target-domain Base fine-tuning uses 100 synthetic examples, batch size 4, and a 5 × 10^-5 learning rate for up to 500 steps.Models are evaluated every 5 steps; all target-domain experiments use a 13,000-token maximum length.Source: §A.1 · p. 13
  • Augmented target-domain settings (Sub, Pert, Mixed) use larger synthetic datasets, batch size 8, and a 1 × 10^-4 learning rate for up to 10 epochs.Evaluation occurs every 50 steps, with a 13,000-token maximum length across target-domain experiments.Source: §A.1 · p. 13

A.2 Training and Inference Cost … D.1 Overview

The sections document the cost and evaluation procedures, structure-preserving subsampling and perturbation methods, and the construction of QUINTD-5 for studying cross-domain data-to-text generation without gold supervision.

A.2 Training and Inference Cost

  • Training and inference experiments run on a single NVIDIA A100 GPU with 80 GB memory; Table 6 reports training memory and time, with DDKD costs measured on OWID as an upper bound.WebNLG-SFT uses Markdown linearisation, which has the largest memory footprint among the considered input formats; zero-shot models incur no training cost.Source: §A.2 · pp. 13–14, Table 6 · p. 14
  • Small distilled models (1.7B / 1B) complete inference on the full 100-example test set within 5 minutes across domains, while teacher-based supervision can take up to 12 hours per domain.Inference uses vLLM with greedy decoding at temperature 0.Source: §A.2 · p. 14

A.3 LLM-as-a-Judge API Details

  • API-based judgments use greedy decoding at temperature 0, a fixed seed of 42, and a 16,384-token generation budget; partial outputs are recovered from the latest syntactically valid JSON.Structured JSON is validated during generation to prevent incomplete or degenerate calls from leaving examples without usable judgments.Source: §A.3 · p. 14

B Subsampling Structural Augmentation Details

  • Subsampling preserves identifiers and internal structure while operating on atomic units; methods enumerate all non-empty subsets or sample subsets at multiple sparsity levels.For Wikidata, the entity identifier is retained while attribute–value subsets are enumerated; Ice Hockey enumerates 255 variants from eight top-level blocks, and GSM Arena selects specification blocks while retaining metadata.Source: B Subsampling Structural Augmentation… · pp. 14–15
  • OpenWeather subsampling restricts fields to core weather data and thins time series to representative daily points, while OWID creates 10 row-subsampled variants at retention ratios from 0.1 to 1.0.OWID variants retain metadata and headers, sample rows deterministically, and restore chronological order.Source: B Subsampling Structural Augmentation… · pp. 14–15

C Noise-based Structural Perturbation Strategies

  • Perturbations vary by format: numeric CSV values are modified within instances, aligned JSON blocks are exchanged across instances, and Markdown property–value pairs are swapped across instances.The strategies aim to preserve structural coherence or validity while introducing variation.Source: C Noise-based Structural Perturbation… · p. 15

D.1 Overview

  • QUINTD-5 provides five times more development instances per domain than QUINTD-1 to compare scaling real inputs with structure-preserving augmentation without gold supervision.It uses the official collection pipelines and identical schemas, distributions, and task definitions, with 500 instances per domain versus 100.Source: §D.1 · p. 15

D.2 Data Analysis … H.2 Example-level and System-level Definitions

QUINTD-5 expands knowledge diversity while largely preserving input characteristics. The appendices also specify the reference-free error-annotation protocol, its evaluation cost, and judge-agreement measures.

D.2 Data Analysis

  • 6,415 vs. 6,401 OpenWeather tokens and 122 vs. 123 Wikidata tokens show QUINTD-5 maintains input lengths while increasing knowledge diversity and preserving structure and task formulation.For Markdown-based Wikidata and CSV-based OWID, added instances introduce entities, predicates, measurements, and values rather than changing structural fields.Source: §D.2 · pp. 15–16, Table 8 · p. 16

E.1 System Prompt

  • The system prompt casts the evaluator as an expert in structured data and units, producing span-level JSON error annotations.It instructs the evaluator to operate correctly with numerical values.Source: §E.1 · p. 16

E.2 User Prompt Template

  • The user template requests a JSON error list with a reason, text span, and type 0–3, sorted by each error’s position in the text.The four types are Incorrect fact, Not checkable, Misleading, and Other.Source: §E.2 · p. 16

D.3 Instance-level Distribution Comparison

  • Figure 3 shows aligned length-distribution modes and spread across four domains; OWID becomes more bimodal in QUINTD-5, reflecting additional extreme cases.Overall, QUINTD-5 increases knowledge diversity while preserving input structure, task formulation, and length distributions, providing a controlled benchmark for comparing real-data scaling with augmentation.Source: §D.3 · p. 16, Figure 3 · p. 17

F Cost of LLM-as-a-Judge Evaluation

  • Approximately $500 was spent on reference-free evaluation across systems, domains, and test instances.Cost is driven by long structured inputs, candidate generations, a detailed error rubric, and the use of two judges, which doubles judgment calls.Source: F Cost of LLM-as-a-Judge Evaluation · pp. 17–18

G Error Types

  • The protocol labels spans as Incorrect Fact, Not Checkable, Misleading, or Other, while excluding omissions and acceptable paraphrases and treating repeated content as one error.Incorrect facts contradict the input; not-checkable claims lack support; misleading spans distort interpretation; Other covers issues such as style, irrelevance, or unnecessary meta-statements.Source: G Error Types · p. 18

H.1 Token-level Agreement

  • For token-level agreement, each judge marks whether each token participates in an error of a given type, then concatenates vectors across outputs from 14 models.The resulting sequences w(A) and w(B) are used to assess agreement between judges.Source: §H.1 · p. 18

H.2 Example-level and System-level Definitions

  • Example-level agreement compares judges’ per-example error counts; system-level agreement compares each system’s mean count, enabling fine-, medium-, and coarse-grained assessment.The counts are computed for each error category, and system means use the number of outputs generated by each system.Source: §H.2 · p. 18

H.3 LLM Judge Agreement Results … I.8 Inter-Annotator Agreement

The human evaluation supports DDKD’s performance among 1.7B models, while agreement is strongest for system-level judgments and weaker for subjective misleading labels and precise error spans. The section also describes how the evaluation set, model selections, annotations, and annotation toolkit were constructed.

I.1 Evaluation Set Construction; I.2 Stratified Sampling with Dual Anchors

  • The input-level evaluation set samples 60 inputs—12 per QUINTD-1 domain—and collects four systems’ outputs for 240 annotations.Sampling targets three instances per error category (Incorrect Fact, Not Checkable, Misleading, Other), using teacher outputs as the primary anchor and zero-shot outputs as fallback when needed; all systems share the same inputs.Source: §I.1 · p. 19, §I.2 · p. 19

I.3 Selection of the best Final Distilled Models.

  • Final distilled-model selection uses GPT-5.1 average errors, error rate, and Gemini-2.5-Pro consistency; GSM Arena uses a ZS-teacher model, unlike the other domains.The selected WebNLG-SFT-teacher models are DDKD-MIXED for Wikidata, Ice Hockey, and OWID, and DDKD-SUB for OpenWeather; GSM Arena selects DDKD-MIXED distilled from the ZS teacher.Source: §I.3 · pp. 19, 21, Table 11 · p. 20

H.3 LLM Judge Agreement Results

  • Across domains, GPT-5.1 and Gemini-2.5-Pro agree at Pearson’s r=0.666 for examples and r=0.955 for systems.The authors take this as support for LLM-based evaluation both for comparing systems and assessing individual outputs.Source: §H.3 · p. 19, Table 9 · p. 20

I Human Evaluation Details; I.4 Human Annotation Protocol

  • Three experienced researchers annotated outputs independently, marking error spans and assigning Incorrect Fact, Not Checkable, Misleading, or Other categories.Annotations record spans, categories, and error counts per category for example-, system-, and domain-level analyses and comparison with LLM judgments.Source: §I.4 · p. 21

I.5 Error Annotation Coverage and Distribution; I.6 Human Annotation Toolkit Interface

  • Error distributions are assessed across five domains, four systems, and both LLM judges; a Streamlit toolkit supports structured input/output inspection and span annotation.Annotators select minimal representative spans, assign an error category, and can provide an explanation and span offset.Source: §I.5 · p. 21, §I.6 · p. 21

I.7 Human Evaluation Results

  • Across evaluated domains, DDKD has the lowest error counts among 1.7B models and the lowest overall normalized average (NormAvg).Domain-level results vary across systems, reflecting differences in domain complexity and error prevalence.Source: §I.7 · p. 21

I.8 Inter-Annotator Agreement

  • IAA uses Krippendorff’s α across token labels, per-example error counts, system-level average counts, and summarisation-quality labels.The methodology uses nominal metrics for token classifications and summarisation labels, and interval metrics for instance and system error counts.Source: §I.8 · pp. 21, 23
  • System-level human agreement is strong (α = 0.90), driven by Incorrect Fact (αsys = 0.81) and Other (αsys = 0.87), while Misleading agreement is near zero.Token-level agreement is fair (α = 0.37); span-boundary ambiguity and subjective Misleading labels do not compromise system rankings.Source: Table 12 · p. 23, §I.8 · p. 23

I.9 LLM-Human Evaluation Agreement … L.2 Compared Source Datasets

This section examines human–LLM evaluation agreement and findings, illustrative outputs, and coverage of distilled models relative to the teacher, including how coverage is scored. It also describes a source-dataset initialization ablation, the datasets compared, and the use of GPT-5.1 coverage scores alongside inter-judge agreement analysis.

I.9 LLM-Human Evaluation Agreement

  • GPT-5.1’s aggregate error-count rankings agree strongly with human rankings (r > 0.90) on GSM-Arena, OWID, and Weather, with positive but weaker correlations on Wikidata (r = 0.83) and Ice Hockey (r = 0.73).The comparison is system-level across four evaluated systems; the table reports Pearson correlations.Source: §I.9 · pp. 23–24, Table 15 · p. 25
  • Incorrect Fact correlations reach r = 0.99 in GSM-Arena and r = 0.95 in OWID, while Misleading correlations vary widely and can be negative, consistent with its subjectivity.Factual-error estimates are less stable in Wikidata, where such errors are rare and the evaluation set is small.Source: §I.9 · pp. 24–25

I.10 Human Evaluation Discussion Summary

  • Human annotators show strong overall agreement (Krippendorff α = 0.578 instance-level; α = 0.902 system-level), but agreement is lower for Not Checkable and Misleading; human and automatic evaluations broadly rank distillation first.Both settings show the lowest performance on OpenWeather and small distilled models outperforming the large teacher on several domains, though evaluation scale and model coverage may explain discrepancies.Source: §I.10 · p. 25

J.1 Wikidata (Entity Description)

J.4 GSM Arena (Product Description)

K.1.1 Prompt

  • Coverage evaluation uses GPT-5.1 and Gemini-2.5-Pro to score completeness; it judges all input facts for Wikidata but task-relevant content for other domains, not exhaustive rows in long tables.Each instance receives a 1–5 score and a 0–1 estimated coverage ratio; normalized mean scores are averaged across domains for cross-system comparison.Source: K Content Coverage Evaluation · p. 26, §K.1.1 · p. 29, §K.1.2 · p. 29

K.1.3 LLM Judge Results

  • Across judges, DDKD-Best remains close to the selected best teacher in coverage and clearly outperforms same-size zero-shot and source-domain SFT baselines.The rankings are consistent for GPT-5.1 and Gemini-2.5-Pro, suggesting gains are not due to systematically omitting content.Source: §K.1.3 · p. 30

K.1.4 Inter-Judge Agreement

  • Pooled GPT-5.1–Gemini-2.5-Pro agreement is high for system-level coverage: Pearson r = 0.959 and Spearman ρ = 0.951 for Score, and r = 0.961 and ρ = 0.959 for Ratio.Instance-level agreement is moderate to strong overall but varies by domain, with lower agreement on Ice Hockey and Weather.Source: §K.1.4 · p. 30, Table 16 · p. 30

L.1 Motivation and Setup

  • The ablation compares WebNLG, E2E, and KELM-Q1 by directly evaluating the same Qwen3-32B teacher on five QUINTD-1 domains, without target-domain distillation or augmentation.It measures faithfulness error numbers (lower is better) and GPT-5.1 content coverage (higher is better).Source: §L.1 · p. 30

K.1.4 Inter-Judge Agreement

  • GPT-5.1 and Gemini-2.5-Pro showed highly consistent rankings; inter-judge agreement reports correlations on per-system mean scores and mean ratios.The table caption labels these correlations rs/ρs and rr/ρr, respectively.Source: §L.1 · p. 30, Table 17 · p. 30

L.2 Compared Source Datasets

  • WebNLG has broad semantic coverage and human-written, fact-aligned references, while E2E has high-quality human references but a narrower restaurant-domain scope.KELM-Q1 is a filtered KELM subset selected using embedding-based and QA-based metrics, but its automatically constructed references are generally less reliable than those of WebNLG and E2E; the filtering follows Song and Gardent (2025).Source: §L.2 · p. 31

L.3 Results

WebNLG is the strongest source-domain initialization for transfer to heterogeneous QUINTD-1 domains, while E2E trades coverage for lower error counts on some domains and KELM-Q1 performs worse overall.

L.3 Results

  • E2E has competitive or slightly lower error counts on Wikidata, Ice Hockey, and GSM Arena, but lower coverage on Ice Hockey (3.99 vs. 2.56) and GSM Arena (3.23 vs. 2.43).The authors suggest E2E's short, simple inputs encourage conservative outputs that omit content when transferred to denser domains.Source: §L.3 · p. 31
  • KELM-Q1 performs worse than WebNLG and E2E across most domains on both error rate and coverage, consistent with noisier, less tightly aligned automatically constructed text.This comparison is qualified by the authors' contrast with human-written WebNLG and E2E datasets.Source: §L.3 · pp. 31–32
  • NE=0.954 and NC=1.000: WebNLG achieves the strongest aggregated initialization trade-off across the five QUINTD-1 target domains.The authors attribute this balance to broad semantic coverage and high-quality, fact-aligned references, which they consider suitable for transferable generation rather than narrow source-domain patterns.Source: §L.3 · pp. 31–32, Table 19 · p. 32

M LLM-as-a-Judge Results

This section compiles LLM-judge evaluations of error types, faithfulness and hallucination-related issues, output length, and content coverage. It reports summary results and domain-specific error analyses, but the supplied passages do not include outcome values.

M LLM-as-a-Judge Results

  • The paper presents Qwen3 and Gemma3 error analyses for Wikidata (Tables 24–31), Ice Hockey (32–39), OpenWeather (40–47), GSMArena (48–55), and OWID (56–63).Source: M LLM-as-a-Judge Results · p. 32
  • The summary tables report judge-based error evaluations for Qwen3 and Gemma3, including macro-averaged error counts and all-category errors across the five QUINTD-1 domains.The captions specify that lower error counts are better; the supplied passages do not show the table values.Source: Table 20 · p. 33, Table 21 · p. 33, Table 22 · p. 34
  • Output-length analysis reports token means and standard deviations over 100 outputs per system for each QUINTD-1 target domain.Lengths use the tokenizer corresponding to the Qwen3 backbone.Source: Table 23 · p. 34
  • Content-coverage sanity checks report mean coverage scores and estimated coverage ratios under GPT-5.1 and Gemini-2.5-Pro judges.Wikidata is assessed for completeness over input facts; other domains are assessed for task-relevant content without exhaustive row matching for long tables.Source: Table 64 · p. 50, Table 65 · p. 50

Related papers

Loading 2608.23391v1…