PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data

Yifei Song, Kun Efimov-Zhang, Claire Gardent

arXiv:2608.23391v1cs.CLcs.AI

TL;DR

The paper asks how to generate faithful text from heterogeneous structured data when target-domain training text and references are unavailable. It distills large teachers into compact students using synthetic target-domain supervision and structure-preserving augmentation. Across five domains, DDKD consistently beats same-size zero-shot and fine-tuned baselines, while augmentation is more effective and cost-efficient than simply scaling real inputs.

  • Problem

    Cross-domain D2T lacks a unified solution when domains, generation goals, and input structures vary and neither in-domain training text nor test references is available.

  • Method

    The paper transfers D2T ability from large teachers to compact students through synthetic target-domain examples, using structural subsampling and perturbation for structure-preserving augmentation.

  • Results

    DDKD consistently outperforms same-size zero-shot and fine-tuned models across five domains; compact distilled models outperform larger LLMs on four of five domains.

  • Takeaways & Limitations

    Cross-domain D2T capability can be transferred to compact models when large LLMs act as intermediate generators, with augmentation mitigating absent target-domain training data.

  • Takeaways & Limitations

    The observed gains are established mainly for Qwen3 and Gemma3 small-scale models, so transferability to significantly larger architectures remains unestablished.

Abstract

from arXiv · show

Structured data exists in many forms (tables, knowledge graphs, charts, and time series), and converting it into text may involve different generation tasks. However, most prior work on data-to-text (D2T) generation has focused on specific tasks and datasets, relying either on task-specific training data or on the zero-shot capabilities of large language models. We study cross-domain D2T generation in a setting where neither in-domain training text nor test references are available, and where domains, generation goals, and input structures vary substantially. We compare data-driven knowledge distillation (DDKD) against zero-shot inference and fine-tuning on out-of-domain D2T data, and introduce structure-preserving augmentation via structural subsampling and perturbation. Experiments on five benchmarks show that, at constant model size (1.7B parameters), DDKD consistently outperforms both fine-tuning and zero-shot inference. Moreover, the resulting small models outperform a much larger finetuned model on two of the five domains, achieving comparable performance on the remaining three. We further construct QUINTD-5, a fivefold extension of QUINTD-1, and show that simply scaling real target-domain inputs yields only modest gains, whereas our augmentation strategy remains more effective and more cost-efficient for cross-domain distillation.

1 Introduction

The paper targets cross-domain data-to-text generation across heterogeneous structures and tasks without in-domain training or reference text. It evaluates data-driven knowledge distillation and finds that compact distilled models transfer more effectively than comparable zero-shot and out-of-domain-finetuned models.

  • Motivation: The setting spans multiple data structures and domains while providing neither target-domain training text nor test references.Challenges include schema and format heterogeneity, long information-dense inputs, and the absence of conventional supervised and reference-based evaluation.
  • Approach: The study compares zero-shot prompting, out-of-domain fine-tuning, and data-driven knowledge distillation for heterogeneous data-to-text tasks.Distilled students learn from synthetic target-domain examples generated by large teachers.
  • Results: DDKD consistently yields lower error rates than same-size zero-shot and WebNLG-finetuned baselines across all five target domains and both model families.The comparison uses Gemma3-1B and Qwen3-1.7B models.
  • Contributions: The paper introduces structure-preserving target-domain augmentation, including structural subsampling and perturbation, to improve robustness to schema and format heterogeneity.It also constructs QUINTD-5 to compare augmentation with scaling real target-domain inputs.
  • Evaluation: The evaluation combines error-taxonomy LLM judges, human validation, inter-judge agreement analysis, and content-coverage checks.The coverage check addresses the possibility that lower error rates arise from conservative under-generation.

2 Related Work

Prior data-to-text work covers many structured inputs but usually concentrates on one or similar structure types and relies on existing training data. This paper combines cross-domain D2T transfer with data-driven distillation in a reference-free setting.

  • Data-to-Text Generation: Prior D2T research has addressed dependency structures, AMRs, knowledge graphs, time series, and tables.
  • Data-to-Text Generation: Most existing work focuses on a single or similar type of structure, limiting coverage of heterogeneous D2T settings.
  • Data-to-Text Generation: Some multi-source approaches handle tables, graphs, and meaning representations, but they rely on existing training data.
  • Positioning: The paper builds on prior evaluation across five domains by studying modeling strategies for those heterogeneous datasets without target-domain reference text.
  • Knowledge Distillation: Recent distillation work transfers rationales, self-generated sequences, domain knowledge, or reasoning abilities, including applications to table-to-text and graph-to-text.

3 Cross-Domain D2T Generation

The framework transfers D2T capability from large teachers to compact students using synthetic target-domain text, while comparing zero-shot, WebNLG fine-tuning, and two DDKD variants. WebNLG supplies broad, human-written source supervision, and DDKD shifts target-domain learning to the student.

  • Setting: The benchmark covers five domains with time series, triples or dictionaries, tables, and charts, and varies in generation objectives and output formats.QUINTD-1 provides development and test inputs without in-domain reference texts.
  • Methods: The study compares zero-shot prompting, WebNLG fine-tuning, and DDKD from either zero-shot or WebNLG-SFT teachers.
  • Methods: WebNLG provides 40K knowledge graph–text pairs as out-of-domain D2T supervision for source initialization.
  • Source Data: WebNLG offers a faithfulness–coverage trade-off that is stronger overall than the alternative source datasets discussed.E2E has competitive error counts on some simpler domains but lower coverage, while KELM-Q1 performs worse on both metrics across most domains.
  • Methods: DDKD trains a compact student on teacher-generated target-domain text pairs, with synthetic supervision expanded through augmentation and perturbation.The student uses maximum-likelihood training on pairs (x, ỹ).

4 Training Data

The training-data pipeline generates synthetic target-domain supervision from QUINTD-1 and expands it with structure-preserving transformations. It contrasts this augmentation with QUINTD-5, which increases real inputs fivefold.

  • Source Preparation: WebNLG graphs are deterministically linearized into JSON, Markdown, and CSV to reduce source–target input-format mismatch.The dataset contains 39,890 graph–text pairs across 16 knowledge categories.
  • Synthetic Data: Synthetic training data are built from QUINTD-1 development inputs to increase quantity and structural diversity while preserving faithfulness.
  • Base Setting: The base setting uses the original 100 development instances per domain and teacher-generated descriptions without augmentation.
  • Subsampling: Structured subsampling creates new instances by selecting non-empty subsets of an input’s attribute–value pairs and verbalizing each subset.
  • Perturbation: Noise-based structural perturbation increases intra-sample variability to reduce overfitting on subsampled data.
  • Combined Augmentation: The mixed strategy combines clean subsampled and perturbed instances, exposing students to diverse inputs and partial structural noise.
  • Real-Data Scaling: QUINTD-5 increases real target-domain inputs from 100 to 500 per domain to compare real-data scaling with augmented supervision.

5 Experiments

The experiments evaluate cross-domain data-to-text strategies across model families, augmentation variants, and reference-free faithfulness protocols. They also test whether observed gains reflect genuine coverage rather than conservative under-generation.

  • Experimental design: Experiments use Qwen3 and Gemma3, with Qwen3 supporting comprehensive augmentation ablations and Gemma3 testing transferability of selected configurations.The study also evaluates QUINTD-5 real-data scaling and performs a content-coverage sanity check.
  • Evaluation: The evaluation targets faithfulness because QUINTD-1 lacks reference texts, using LLM judges to compare outputs directly against structured inputs.Reference-based metrics such as BLEU and ROUGE are therefore inapplicable.
  • Validation: Coverage checks test whether lower error rates could instead result from shorter or less informative outputs.The check uses the same models and two LLM judges as the human-evaluation subset.
  • Evaluation: Average errors per output are aggregated across four error types: Incorrect Fact, Not Checkable, Misleading, and Other.Judges identify erroneous spans and assign potentially co-occurring error categories.
  • Evaluation: GPT-5.1 provides the main results, while Gemini-2.5-Pro independently evaluates systems to assess robustness under heterogeneous schemas and long inputs.Agreement is analysed separately by target domain and error category.
  • Human validation: Human evaluation covers 240 outputs from four representative systems, comparing small zero-shot, fine-tuned, distilled, and large-teacher models.Annotators label example-level error presence, severity, and spans on a diagnostic subset.

6 Results and Discussion

Across five domains, distilled 1.7B models outperform same-size zero-shot and fine-tuned baselines, while augmentation improves distillation beyond simply adding real inputs. Results vary by domain, and larger models retain an input-faithfulness advantage in some comparisons.

  • Overall results: At constant size, 1.7B distilled models consistently outperform fine-tuning and zero-shot prompting.A compact distilled model also outperforms Qwen3-32B and GPT-4.1 on four of five domains.
  • Real-data scaling: NormAvg is 0.42 versus 0.27 for 500-real and best augmented models under the ZS teacher, and 0.20 versus 0.10 under the WebNLG-SFT teacher.Lower values indicate fewer normalized errors.
  • Augmentation: Augmentation improves distillation quality: perturbation alone or mixed subsampling and perturbation helps on four of five datasets.The strategy is less effective for weather forecasting, which requires strong aggregation and summarisation.
  • Domain differences: Weather consistently performs worst, while the rankings of Wikidata, GSM Arena, Ice Hockey, and OWID vary with the training approach.The weather task summarizes 15 data points, whereas teacher choice changes the ordering of the other domains.
  • Teacher effects: GSM Arena favors a zero-shot teacher over the WebNLG-SFT teacher, and the selected DDKD model is therefore distilled from the zero-shot teacher.The paper links this difference to the domain’s entity-centric consumer-device knowledge.
  • Error analysis: The best SFT-distilled small model has 0.05 Not Checkable errors, versus 0.53 for zero-shot and 0.70 for fine-tuned models.These are macro-averaged hallucination-related error counts per output across target domains.
  • Limitation: Large models remain more input-faithful overall, with Incorrect errors of 0.14, 0.50, and 0.41 versus 1.13 for the best distilled model.The comparison indicates that distillation still needs improvement in conditioning on input structure.
  • Validation: DDKD achieves higher coverage than same-size baselines while remaining close to the best teacher, and system-level judge agreement exceeds r > 0.95.Human evaluation shows strong system-level agreement with GPT-5.1 and broadly consistent rankings.

7 Conclusion

The paper finds that compact models can acquire cross-domain data-to-text capabilities when large LLMs serve as intermediate generators. Data augmentation and data-driven knowledge distillation help address missing target-domain training data.

  • Conclusion: Data augmentation and data-driven knowledge distillation help mitigate the absence of target-domain training data.The conclusion connects these methods to cross-domain generation across multiple input structures and objectives.
  • Conclusion: Cross-domain data-to-text generalization can be transferred to compact models by using large LLMs as intermediate generators rather than deployment targets.The conclusion frames this as a way to study new generation tasks from existing real-world data.

8 Limitations

The study’s limitations concern model coverage, heuristic augmentation design, and the cost and potential domain sensitivity of LLM-based judging.

  • Model coverage: Gains are evaluated primarily on Qwen3 and Gemma3 small models, so transferability to significantly larger architectures remains unestablished.The focus targets small, long-context models because long-sequence fine-tuning, inference, and synthetic data generation are computationally expensive.
  • Model and evaluator coverage: Closed-source GPT-4.1 and GPT-5.1 models are used for comparison and evaluation, while additional open-weight families and open evaluators remain future directions.They are not used for training-data generation, supervision, or distillation, preserving auditability of the training pipeline.
  • Heuristic design in structural augmentation: Structural augmentation relies on heuristic choices such as defining atomic units and enforcing exchangeability constraints.These choices impose minimal structural assumptions and may not be optimal for every domain.
  • Cost and bias of LLM-as-a-Judge: LLM-based judging can be domain-sensitive and costly, despite mitigation through two strong judges and corroborating human evaluation.The authors expect open-model metrics to reduce cost and improve accessibility.

A.2 Training and Inference Cost

The appendix describes the computational setup and the structure-preserving procedures used to construct augmented inputs across domains. It also defines the benchmark extension used to compare augmentation with real-data scaling.

  • Computational setup: All training and inference experiments use a single NVIDIA A100 GPU with 80 GB memory.Training-cost tables report peak memory and wall-clock time under this setup.
  • Training cost: DDKD costs are measured on OWID to represent the longest target-domain inputs and an upper bound on resource requirements, while zero-shot models have no training cost.WebNLG-SFT uses Markdown linearisation, which has the largest memory footprint among the considered formats.
  • Inference cost: Small distilled models complete inference on the full 100-example test set within five minutes across domains, whereas generating one domain’s synthetic supervision with large teachers can take up to 12 hours.Large-teacher generation is substantially more expensive under long-context settings.
  • Structural subsampling: Structural subsampling preserves internal structure by operating on atomic units while retaining domain identifiers and high-level metadata.Depending on granularity and complexity, the method enumerates all non-empty subsets or samples subsets at multiple sparsity levels.
  • Domain-specific construction: The subsampling scheme is domain-specific: Ice Hockey enumerates 255 non-empty subsets of eight top-level blocks, while OpenWeather thins time series to representative daily points.Other domains subsample specification blocks or date,value rows while preserving metadata and headers.
  • QUINTD-5: QUINTD-5 provides five times more development instances per domain than QUINTD-1, enabling controlled comparison of real-input scaling and structure-preserving augmentation.The extension uses identical schemas, distributions, and task definitions, differing only in data volume.

D.2 Data Analysis

QUINTD-5 preserves QUINTD-1’s input structure and length characteristics while expanding knowledge diversity, enabling controlled analysis of real-data scaling. The evaluation also uses reference-free, span-level error annotation supported by LLM judges and human validation.

  • QUINTD-5 data properties: QUINTD-5 preserves similar input length distributions while substantially increasing knowledge diversity.Average input lengths remain stable after the fivefold data increase, while distinct values and entities grow substantially.
  • Instance-level distributions: Across four domains, QUINTD-1 and QUINTD-5 have closely aligned instance-level length distributions.OWID is an exception, showing a more pronounced bimodal pattern associated with additional extreme cases.
  • Scaling effects: Scaling real target-domain data increases coverage and knowledge diversity but can introduce greater distributional heterogeneity.The OWID bimodal pattern indicates additional extreme cases that were underrepresented in QUINTD-1.
  • Instance-level distributions: Density normalization permits direct KDE comparison despite QUINTD-1 and QUINTD-5 having 100 and 500 development instances.Figure 3 compares prompt token-length distributions across five domains.
  • Evaluation setup: The error taxonomy distinguishes incorrect facts, not-checkable statements, misleading phrasing, and other problematic content.Errors are annotated over minimal representative spans, while omissions and acceptable paraphrases are not penalized.

H.3 LLM Judge Agreement Results

The paper assesses agreement among LLM judges and human annotators at multiple aggregation levels. Agreement is strongest for system-level comparisons, supporting model ranking, while token- and category-level judgments remain more subjective.

  • LLM judge agreement: Pearson’s r=0.666 at the example level and r=0.955 at the system level across all domains.The higher system-level agreement supports robust comparison of data-to-text systems.
  • Human validation: Human evaluation covers 60 inputs and 240 outputs from zero-shot, fine-tuned, distilled, and large-teacher systems.Inputs are stratified across five domains and error categories, with all systems evaluated on the same inputs.
  • Agreement protocol: The agreement protocol evaluates consistency at token, example, and system levels within the same domain and task.Separate correlations are computed across domains and error categories.
  • Human validation: Across evaluated domains, DDKD has the lowest error counts among 1.7B models and the lowest normalized average overall.Domain-level variation remains across systems.
  • Human validation: Human annotators show strong system-level agreement at α=0.90, driven by Incorrect Fact αsys=0.81 and Other αsys=0.87.The Misleading category has near-zero agreement, reflecting its subjectivity.
  • Human validation: Token-level human agreement is fair at α=0.37 because misleading judgments and exact span boundaries are subjective.Despite this limitation, high system-level agreement preserves the validity of final model rankings.

I.9 LLM-Human Evaluation Agreement

The study validates GPT-5.1 as a scalable proxy for system-level human evaluation, while showing that agreement varies substantially by domain and error category. Human evaluation also broadly supports the automatic ranking trends.

  • Agreement results: r > 0.90 system-level agreement in All Errors was observed for GSM-Arena, OWID, and Weather, with positive but weaker alignment for Wikidata and Ice Hockey.Correlations were r = 0.83 for Wikidata and r = 0.73 for Ice Hockey.
  • Agreement results: r = 0.99 for Incorrect Fact errors in GSM-Arena and r = 0.95 in OWID indicate especially strong GPT-5.1 alignment for factual hallucinations.Correlations were weaker for Wikidata, where factual errors were rare and estimates were less stable.
  • Human agreement: Krippendorff α = 0.902 at the system level exceeded instance-level agreement of α = 0.578 among the three human annotators.Incorrect Fact and Other showed high agreement, whereas Not Checkable and Misleading were more subjective.
  • Interpretation: GPT-5.1 reliably reproduces human system rankings at the system level, supporting its use as a scalable evaluation proxy.The evaluation aggregates average errors per system because token-level or span-level agreement is infeasible under the annotation setup.
  • Cross-evaluation consistency: Automatic and human evaluations both rank knowledge distillation first overall, although discrepancies remain in domain rankings and teacher–student gaps.The human evaluation covers one distilled model per domain, whereas automatic evaluation includes multiple distilled variants.

L.3 Results

WebNLG is the strongest source-domain initialization for transfer to heterogeneous QUINTD-1 domains. Although other datasets can reduce errors on simpler domains, their outputs may omit more content or provide noisier supervision.

  • Source-domain initialization: NE=0.954 and NC=1.000 make WebNLG the strongest aggregated initialization under normalized error and coverage summaries.The result indicates the most favorable trade-off between factual correctness and content coverage across QUINTD-1 domains.
  • Source-domain initialization: E2E achieves competitive or slightly lower error counts than WebNLG on Wikidata, Ice Hockey, and GSM Arena, but with substantially lower content coverage.Coverage differences are 3.99 vs. 2.56 on Ice Hockey and 3.23 vs. 2.43 on GSM Arena.
  • Source-domain initialization: KELM-Q1 performs worse than WebNLG and E2E across most domains on both error rate and coverage.The passage attributes this pattern to noisier and less tightly aligned supervision from automatically constructed text.
  • Conclusion: WebNLG provides the best balance between broad semantic coverage and fact-aligned references among the compared source datasets.This supports its use for learning transferable data-to-text capabilities rather than narrow source-domain patterns.

M LLM-as-a-Judge Results

The evaluation reports error summaries, output lengths, domain-specific error analyses, and content coverage results for Qwen3 and Gemma3 systems across the five QUINTD-1 domains.

  • Result summaries: Tables 20–23 summarize Qwen3 and Gemma3 results, including error categories and output length analysis.The summaries cover aggregate performance across the target domains.
  • Coverage analysis: Tables 64–65 report content coverage across domains as a complementary evaluation dimension.Coverage is summarized separately from the faithfulness-related error analyses.
Loading 2608.23391v1…