PaperlayerView exact revision on arXiv

Source-linked AI summary

XQDT: eXplainable and Quantitative Data-Text Alignment Metric with Feedback Signals

Kun Efimov-Zhang, Yifei Song, Claire Gardent

arXiv:2608.29948v1cs.CL

TL;DR

Existing data-text alignment metrics offer limited explanations, while prompted LLM judges can require expensive evaluation. XQDT fine-tunes a verifier to identify localized alignment errors, aggregates them into precision, recall, and F1, and uses the outputs for refinement. Across multiple datasets and settings, it is presented as effective for evaluation and downstream correction, with larger gains for text-to-data extraction than data-to-text generation.

  • Problem

    Reliable bidirectional evaluation is needed to determine whether structured data and text faithfully express the same information, but existing metrics provide limited explanations and some LLM-based methods require multiple expensive calls.

  • Method

    XQDT fine-tunes a Transformer-based generative verifier to label Correct, Omitted, Incorrect, and Extra units, then aggregates these local judgments into precision, recall, and F1.

  • Results

    Across multiple datasets, settings, and revision models, XQDT is effective for evaluation and feedback-driven refinement, with substantial gains for text-to-data extraction and smaller measurable improvements for data-to-text generation.

  • Takeaways & Limitations

    XQDT supports alignment-oriented assessment and can function as a practical feedback component in downstream correction and refinement pipelines.

  • Takeaways & Limitations

    The triple-based formulation transfers less directly to richer structures such as AMR, dependency structures, and discourse graphs, where meaning spans multiple components.

Abstract

from arXiv · show

Evaluating data-text alignment remains challenging: existing metrics often provide limited explanations for the scores, while prompt-based LLM-as-Judge methods can be expensive and unreliable. We present an end-to-end explainable evaluation metric that fine-tunes a language model to identify omitted, extra, incorrect, and correct data units in a data-text pair. These local judgements are aggregated into precision, recall, and F1 scores, providing both fine-grained diagnostic feedback and an interpretable measure of alignment quality. Across benchmarks, our fine-tuned models outperform LLM-as-Judge methods in error prediction and achieve competitive precision, recall, and F1 scores, while maintaining strong correlation with human judgements. Beyond evaluation, our verifier outputs also provide useful feedback signals for downstream correction and refinement, supporting alignment-oriented improvement of data-to-text and text-to-data. Code and resources are available at https://github.com/guihuzhang/xqdt.

1 Introduction

XQDT addresses the need for bidirectional, explainable data-text alignment evaluation by combining localized error labels with quantitative scores. Its verifier is evaluated across datasets and supports downstream correction and refinement.

  • XQDT evaluates consistency between structured data and English text in both data-to-text and text-to-data settings.
  • Unlike methods that primarily produce scores or require multiple expensive LLM calls for explanations, XQDT predicts omitted, extra, incorrect, and correct labels for individual data units.
  • These localized predictions are aggregated into precision, recall, and F1 scores, combining fine-grained explanations with quantitative evaluation.
  • XQDT is evaluated against ground-truth error labels and human annotations across KELM, WebNLG, and E2E in synthetic, generated, and manually validated settings.
  • The verifier outputs can provide feedback signals for downstream correction and refinement in both data-to-text and text-to-data.

2 Related Work

Prior data-text alignment metrics use varied entailment, question-answering, triple-level, regression, or LLM-based strategies, but differ in localization and cost. XQDT instead anchors unit-level judgments in structured input while discovering unsupported text content.

  • Prior alignment metrics include NLI entailment, question generation and answering, triple-level recall scoring, direct regression, and prompted or distilled LLM evaluation.
  • DQE uses multi-step question generation and answering, FactSpotter emphasizes recall, and MonoLR predicts aggregate scores without explicit unit-level localization.
  • Atomic factuality metrics decompose text into facts for verification, whereas XQDT targets bidirectional alignment with a given structured input.
  • XQDT anchors Correct, Omitted, and Incorrect units in the input, while Extra units are discovered and localized from the text.
  • Alignment signals have also been studied as feedback for iterative refinement and optimization of data-to-text and text-to-data systems.

3 Method

XQDT fine-tunes a generative Transformer to label local data-text differences, then aggregates those labels into precision, recall, and F1. Synthetic perturbations create plausible error cases from WebNLG and E2E data-text pairs.

  • Given a data-text pair, XQDT labels triples as Correct, Incorrect, Omitted, or Extra according to their alignment status.
  • Precision, recall, and F1 are computed by aggregating labels on individual data units.
  • Incorrect units lower both precision and recall, Omitted units affect recall, and Extra units affect precision.
  • The experiments use WebNLG, KELM, and E2E, with aligned pairs supporting synthetic construction and KELM also serving as naturally noisy silver data.
  • The XQDT evaluator is a Transformer-based generative model fine-tuned on synthetic (D, T, E) examples containing labeled differences.
  • Data Construction: Synthetic data are created by perturbing aligned WebNLG and E2E pairs while preserving semantic relatedness, producing plausible Correct, Incorrect, Omitted, and Extra cases.
  • Data Construction: Omitted cases extend a gold data instance with related existing data or fabricated triples absent from the text.
  • Data Construction: Extra cases remove triples from the data while retaining text or concatenate related texts while retaining one graph, labeling text-only triples as Extra.

4 Experiments

XQDT is evaluated for fine-grained error detection, quantitative agreement with human judgements, coarse-label prediction, cross-dataset transfer, and auditing noisy data-text pairs. Across these settings, fine-tuned verifiers generally outperform prompted judges, while performance varies by error type, dataset, and model size.

  • Evaluation setup: XQDT is evaluated using synthetic error-label F1, correlations with human precision/recall/F1 judgements, coarse E2E labels, cross-dataset transfer, and KELM auditing.The experiments cover WebNLG, E2E, and KELM, with synthetic, human-annotated, and noisy data-text pairs.
  • Synthetic error detection: Fine-tuned XQDT models substantially outperform prompted approaches across all error types and metrics on both WebNLG and E2E.GPT-5.1 is the strongest prompted model but remains below the fine-tuned models.
  • Synthetic error detection: EXTRA is the most challenging error type because it requires discovering and decomposing unsupported information from text rather than copying triples from the input data.OMITTED and INCORRECT errors can be identified against input triples, whereas EXTRA content is absent from the structured input.
  • Human-judgement correlation: Qwen3-8B achieves the highest correlations with 4L-RP-Human precision, recall, and F1 annotations, outperforming prompted GPT-5.1.Recall correlations are more consistent than precision correlations, and model size does not consistently predict agreement with human judgements.
  • Human-judgement comparison: On E2E coarse labels, missing detection is consistently strong, whereas added detection is markedly weaker alongside very low inter-annotator agreement for the added label.E2E collapses XQDT’s fine-grained labels into is-ok, has-missing, and has-added categories.
  • Cross-dataset generalisation: WebNLG-only verifiers transfer to E2E: 4B and 8B models keep is-ok and has-missing F1 within 0.6 points and text-level correlations within 1.0 point across training settings.For Extra errors, strict F1 is much lower than similarity F1, which remains above 80.0; Omitted and Incorrect triple-level F1 exceed 92.0.
  • Noisy-data auditing: On KELM auditing, Qwen3-8B has the highest precision and F1, while GPT-5.1 has the highest recall; Qwen3-8B offers a more balanced trade-off.KELM contains approximately 18M automatically generated data-text pairs, making targeted verification useful for selecting or post-processing instances.

5 Using XQDT for Data-to-Text and Text-to-Data Improvement

XQDT verifier outputs are converted into revision feedback for iterative improvement of both text-to-data and data-to-text systems. The gains are larger for text-to-data than for data-to-text.

  • Feedback mechanism: The verifier identifies omitted, extra, and incorrect units, then provides them as feedback to a revision LLM for iterative correction.For text-to-data, the revision model adds, removes, or revises triples; for data-to-text, it revises text while preserving source-supported content.
  • 5.1 Text-to-Data Improvement: Up to 9.9 points on GenWiki and 6.6 points on WebNLG, XQDT improves Exact F1 over unrevised text-to-data outputs.On WebNLG, XQDT performs best across representative BT5 and ReGen settings.
  • 5.2 Data-to-Text Improvement: XQDT feedback yields smaller but measurable improvements for data-to-text generation on WebNLG.The experiments use Control Prefix as the base generator and report representative and full results across repair models.

6 Conclusion

XQDT evaluates bidirectional data-text alignment through localized error types and aggregate scores, then reuses those verifier outputs for downstream correction. Experiments support its usefulness for assessment and refinement, with larger gains for text-to-data than data-to-text.

  • 6 Conclusion: XQDT combines fine-grained error analysis with quantitative precision, recall, and F1 scores for bidirectional data-text alignment.Its localized error reports identify sources of mismatches and suggest directions for model improvement.
  • 6 Conclusion: Verifier outputs improve both data extraction and text generation, producing substantial gains for text-to-data and smaller measurable gains for data-to-text.The same alignment-oriented verification signals can be used in downstream correction and refinement pipelines.

Limitations

The framework has scope, measurement, computational, task-comparability, and feedback-sensitivity limitations. Its triple-centered design is best suited to graph-to-text and text-to-graph settings.

  • Representation scope: Triple-based atomicity limits direct transfer to richer structures such as AMR, dependency structures, and discourse graphs.In those representations, local substitutions or merges may not produce equally clear and controlled alignment deviations.
  • Measurement granularity: Data-item-level precision, recall, and F1 can differ systematically in absolute scale from finer-grained human judgements that allow partial credit.A triple with only one or two incorrect elements is counted as entirely incorrect by the framework.
  • Computational cost: Structured unit-level reports can make inference more computationally demanding than metrics returning a single scalar score.Worst-case cost increases with the number of error units, especially when text contains much unsupported content.
  • Downstream scope: Downstream feedback utility is task-dependent, so improvement gains across text-to-data and data-to-text should not be directly compared.Triple edits map naturally to structured correction, whereas data-to-text requires free-form text revision.
  • Feedback sensitivity: Verifier misjudgements can induce unnecessary edits when outputs are consumed as revision feedback.Iterative revision may correct some earlier mistakes, but downstream edits remain sensitive to verifier errors.

D Data Source Description

The paper evaluates XQDT using several data-text resources, synthetic perturbations, human judgements, and prompted LLM baselines. These materials cover error detection, metric correlation, and downstream alignment improvement.

  • Source datasets: WebNLG, KELM, and E2E provide complementary data-text benchmarks spanning verified graph verbalizations, synthetic Wikidata pairs, and restaurant-domain meaning representations.The experiments use WebNLG 3.0, KELM, and the cleaned E2E release.
  • E2E: E2E contains 10,852 cleaned distinct meaning representations paired with 42,517 reference texts and human annotations for 630 test meaning representations.Its annotations include coarse semantic labels plus overall quality and naturalness scores.
  • Synthetic data: Synthetic error-detection data are built from aligned WebNLG and E2E pairs by perturbing them into omitted, incorrect, and extra cases while preserving semantic plausibility.A reference-quality filter and a manual audit of 180 examples further assess the synthetic data.
  • Human evaluation: Human-correlation analyses compare verification precision, recall, and F1 with targets derived from WebNLG and 4L-RP-Human annotations.The study reports Pearson, Spearman, and Kendall correlations, including human precision derived as correctness multiplied by relevance and recall from coverage.

I Further Analysis on E2E Human Annotations

The E2E analysis evaluates XQDT against human coarse error labels and studies annotation reliability, model scaling, and human-rating correlations. It finds strong performance on missing-content judgments but weak agreement and classification for added content.

  • Evaluation setup: E2E annotations are converted into is-ok, has-missing, and has-added tasks for evaluating XQDT and baseline metrics.XQDT maps omitted or incorrect units to has-missing and extra units to has-added.
  • Reliability: The added task has only slight inter-annotator agreement (α=0.104), explaining its considerably weaker classification results.Agreement for ok and missing is stronger, and outlier annotations are removed before correlation analysis.
  • Human-rating correlation: At the text level, XQDT verifiers are consistently stronger than prior baselines across Pearson, Spearman, and Kendall correlations with E2E human quality ratings.Recall-based scores generally align better with ratings because annotators agree more on missing content than additions.
  • Human annotation: The KELM study samples 60 diverse examples across graph sizes and four Qwen3 model scales, producing 240 model outputs for annotation.Three annotators labeled overlapping and independently assigned samples; overlap agreement reached α=0.573 for verification and α=0.646 for missed-error supplementation.
  • Annotation workflow: The annotation workflow first verifies model predictions, then supplements missed incorrect, omitted, and extra content.Extra content is copied from text fragments absent from the data, while omitted and incorrect items are copied from input triples.

L Further Consistency Analysis with Human Annotations on Noisy KELM

The noisy-KELM analysis measures whole-pair consistency and fine-grained error detection across model scales. Performance improves through 8B, while false alarms, phrasing variation, and subtle relational distinctions remain important weaknesses.

  • Binary consistency: The Qwen3-8B model achieves the best binary-consistency F1 of 75.5% and accuracy of 78.3%, while the 14B model underperforms 4B.The result indicates that larger scale alone does not guarantee better calibration.
  • Error analysis: False alarms comprise 56–75% of mistakes across models, identifying error-reporting calibration as the primary bottleneck.The challenge is deciding when to flag an error rather than only assigning its type or target.
  • Failure modes: Training on surface-mirroring verbalisations can cause XQDT to flag semantically faithful but more loosely phrased expressions as errors.A reported example labels a faithful administrative-location paraphrase as Incorrect, although annotators call it a false alarm.
  • Failure modes: Subtle relational distinctions can produce missed errors, such as confusing nominated with received in an award statement.This failure concerns semantic relation differences rather than simple lexical mismatch.
  • Error-type results: On Missing errors, the 0.6B model has high recall but very low precision, whereas the 14B model has perfect Extra precision but low recall.These contrasting patterns show different precision–recall trade-offs across error types.
  • Scaling with input complexity: The 8B model performs best on instances with 2–5 triples and remains close to 4B at six triples, while 0.6B and 14B degrade more.The degradation indicates limited robustness as graph complexity grows.

M WebNLG-to-E2E Cross-Dataset Transfer

The transfer study evaluates WebNLG-trained verifiers on E2E without further training and compares them with jointly trained models. Transfer is most consistent for 4B and 8B models, while fine-grained Extra matching is more representation-sensitive.

  • Transfer setup: WebNLG-only Qwen3 verifiers are evaluated on E2E without further training against models trained jointly on WebNLG and E2E.Both settings use the same E2E examples.
  • Human-score correlation: For 4B and 8B models, text-level correlation differences between WebNLG-only and joint training are at most 1.0 point.The smaller WebNLG-only models show a marked text-level reduction.
  • Coarse-label transfer: On E2E coarse-label tasks, 4B and 8B is-ok and has-missing F1 scores differ by no more than 0.6 points between training settings.Has-added performance remains relatively low in both settings, consistent with low inter-annotator agreement.
  • Uncertainty analysis: Paired bootstrap intervals show that WebNLG-only and joint training are not reliably distinguishable for Qwen3-8B.For Qwen3-4B, small differences remain significant for is-ok and has-missing.
  • Fine-grained localisation: Transferred 4B–14B models exceed 92.0 strict F1 for both Omitted and Incorrect errors.Extra strict F1 is lower, but similarity F1 reaches 80.1–84.0 because semantically corresponding E2E units may be expressed as WebNLG-style triples.
  • Overall transfer: Overall, 4B and 8B transfer retains coarse-label performance, text-level correlations, and strong Omitted and Incorrect localisation close to joint training.The cross-dataset result is weaker for Extra errors and smaller models.

N Detailed Precision, Recall, and F1 for Error Detection

The detailed evaluation compares triple-, element-, and similarity-based error metrics across Omitted, Extra, and Incorrect categories. Fine-tuned verifiers outperform prompted judges in precision and recall, with gains that generally plateau beyond 8B.

  • Evaluation metrics: Evaluation uses triple-level, element-level, and similarity-based metrics to assess error-prediction quality from complementary perspectives.Triple-level matching requires exact subject, predicate, and object agreement.
  • Similarity evaluation: Similarity scoring uses BGE-1.5-Large embeddings and Hungarian one-to-one matching to award partial credit for semantically close predicted and gold triples.Precision, recall, and F1 are computed from matched cosine similarities without allowing one gold triple to match multiple predictions.
  • Aggregation: Scores are reported separately for Omission, Extra, and Incorrect errors, with macro-average equal to the arithmetic mean of the three per-class scores.Correct no-error predictions receive perfect scores, while false alarms on positive pairs receive zero scores.
  • Scaling results: Performance increases with model size up to 8B, while further scaling typically yields only marginal improvements.This pattern is reported across precision and recall tables for both WebNLG and E2E.
  • Baseline comparison: Fine-tuned verification models consistently outperform prompted LLM-as-judge baselines on both precision and recall, despite the baselines using much larger language models.The comparison spans both WebNLG and E2E data.

O Details for Text-to-Data and Data-to-Text Improvement

The appendix details verifier-feedback repair experiments for text-to-data and data-to-text systems, including prompts, full model-wise results, and representative qualitative examples. Across text-to-data settings, XQDT is strongest on stricter Exact and Strict criteria in most cases, while some Partial F1 gains do not consistently indicate factual correction.

  • Repair prompts: Repair prompts revise generated text or extracted triples using source information and verifier feedback, with revision experiments using temperature = 0.7 and seed = 42.The data-to-text prompt revises previously generated text from source triples; the text-to-data prompt revises extracted triples from source text.
  • Text-to-data repair: XQDT remains strongest on Exact and Strict criteria in most text-to-data repair settings across WebNLG and GenWiki.The full results cover all evaluated repair models, three repair rounds, and BT5 and ReGen base extractors.
  • Text-to-data repair: Partial F1 occasionally narrows the gap for NLI and PiVe on GenWiki, but these gains generally do not transfer to Exact or Strict F1.The appendix interprets this pattern as increased local overlap rather than consistently better factual correction in outputs.
  • Data-to-text repair: Data-to-text repair results on WebNLG are reported across three repair rounds for Control Prefix and a broad set of automatic metrics.Reported metrics include BLEU, ROUGE, METEOR, chrF++, PARENT-F1, BERTScore-F1, and SBERT.
  • Qualitative examples: Representative examples show how verifier feedback changes data-to-text and text-to-data outputs across rounds and common error types.The examples are presented in Figures 6–9 and focus on visually interpretable repairs.
  • Error detection results: The appendix provides WebNLG and E2E error-detection precision, recall, and F1 tables, including triple-level, element-level, similarity-based, and macro-averaged scores.WebNLG results span Extra, Omission, and Incorrect errors; E2E results report the same error categories for verification models.
Loading 2608.29948v1…