PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing Quality of Experience in Natural Language Generation of German Text

Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz, Sebastian Möller

arXiv:2608.18888v1cs.CL

TL;DR

Traditional automatic metrics provide limited evidence about human-perceived NLG quality, motivating a user-centered evaluation framework for German text. The paper introduces TextQ-German, collects QoE ratings for summarization and translation, and evaluates prediction models, finding strong performance from hybrid and linguistic-feature approaches with held-out validation evidence.

  • Problem

    Traditional automatic metrics often fail to capture the multifaceted, user-perceived quality of generated text.

  • Method

    The paper builds TextQ-German from crowdsourced German-speaker ratings for summarization and translation, derives task-specific perceptual dimensions, and trains transformer-based, linguistic-feature-based, and hybrid QoE predictors.

  • Results

    Hybrid models outperform pure transformer baselines in most experiments, while selected linguistic features approach fine-tuned language-model performance; held-out validation supports generalization for ATS and overall MT QoE.

  • Takeaways & Limitations

    TextQ-German provides a publicly accessible human-centered resource and baseline models for evaluating German NLG quality from a QoE perspective.

  • Takeaways & Limitations

    The study models text-observable attributes but omits contextual QoE factors such as task utility, prior expectations, and emotional response, while its relatively small dataset limits training and generalizability.

Abstract

from arXiv · show

The rapid advancement of Natural Language Generation (NLG) has made the reliable evaluation of generated text increasingly critical, as these systems, such as large language models (LLMs), are now widely deployed in real-world applications. However, traditional automatic metrics fail to capture the multifaceted nature of perceived quality. In this paper, we introduce TextQ-German, a novel dataset suite for human-centered evaluation of German NLG from a Quality of Experience (QoE) perspective, covering automatic text summarization and machine translation. Through crowdsourcing studies with German speakers, we collect human quality ratings and identify relevant perceptual quality dimensions for each task. We develop automatic QoE prediction models, including transformer-based, linguistic feature-based, and hybrid approaches. Hybrid models outperform pure transformer baselines in almost all experimental settings, while linguistic features alone can approach the performance of fine-tuned language models. The dataset is extended with LLM-generated outputs annotated with overall QoE scores. Final validation on held-out sets indicates generalization to unseen data. Our work contributes a publicly accessible resource for NLG evaluation and baselines for automatic QoE prediction, providing a foundation for developing NLG systems that better align with human quality perception.

1 Introduction

Reliable NLG evaluation is difficult because conventional automatic metrics often diverge from human judgments and overlook users’ subjective experience. The paper addresses this gap for German by introducing a user-centered QoE dataset and prediction models for summarization and translation.

  • Evaluation gap: BLEU and ROUGE are efficient but often correlate poorly with human judgments because they emphasize lexical overlap over structure and semantic context.Their surface-level comparisons are limited for nuanced evaluation.
  • Quality of Experience: Quality of Experience captures users’ subjective perceptions, including expectations, context, and emotional responses, alongside technical properties.This perspective evaluates generated text from the end user’s viewpoint.
  • Scope and contribution: The study targets comparatively underexplored German NLG evaluation across Automatic Text Summarization and Machine Translation.It examines both fine-grained perceptual dimensions and overall perceived quality.
  • Scope and contribution: TextQ-German provides human-annotated QoE ratings and automatic prediction models intended to evaluate unseen generated text.The work derives task-specific perceptual dimensions from user ratings rather than prespecifying fixed criteria.
  • Scope and contribution: The work offers a human-oriented benchmark that moves beyond traditional metrics for assessing modern NLG systems.The article also consolidates and extends prior project publications, with datasets and additional data publicly accessible.

2 Related Work

Related work shows that NLG evaluation combines efficient automatic metrics with human judgments, while German research has focused mainly on readability and complexity. QoE-based text evaluation extends this line toward overall perceived quality and task-specific perceptual dimensions.

  • NLG evaluation: Reference-based metrics such as BLEU and ROUGE are efficient and reproducible but correlate weakly or inconsistently with human ratings.Their limitations motivate learned and human-centered evaluation approaches.
  • NLG evaluation: Human evaluation treats text quality as multifaceted, but definitions and procedures for dimensions such as fluency, adequacy, coherence, and relevance vary across studies.Prior work has documented substantial inconsistency in quality-criterion terminology.
  • QoE and subjective prediction: QoE research commonly converts subjective judgments into Mean Opinion Scores or ratings of multiple perceptual dimensions for automatic prediction.This paradigm is established in speech, audio, image, and video services.
  • QoE and subjective prediction: TextQ-German extends German text QoE evaluation beyond readability by collecting overall and attribute-level ratings for Automatic Text Summarization and Machine Translation.These ratings support empirical derivation of task-relevant perceptual dimensions.
  • German text evaluation: German text research has primarily modeled readability and complexity from linguistic characteristics and contextual representations.GermEval 2022 further framed German text-complexity prediction as a regression problem combining transformer representations with textual statistics.

3 Dataset Resource

TextQ-German is a publicly released resource for evaluating German NLG through user-perceived quality, covering summarization and translation. It combines human QoE ratings, task-specific perceptual dimensions, LLM-based extensions, validation sets, and streamlined evaluation procedures.

  • Dataset Resource: TextQ-German supports analysis of perceptual dimensions, automatic QoE prediction, and robustness testing beyond non-LLM generation.Its resources include linguistic-feature-based, transformer-based, and hybrid prediction settings, plus LLM-based extensions and final validation sets.
  • Dataset Resource: The suite covers automatic text summarization and machine translation through six subsets spanning dimension-level ratings, LLM-generated extensions, and held-out validation sets.These subsets support both fine-grained perceptual dimensions and overall quality scores across classical and LLM-based generation scenarios.
  • Dataset Resource: German-speaking crowdsourcing studies used bipolar adjective pairs to identify, validate, and quantify task-specific perceptual quality dimensions.The studies applied Semantic Differential scaling, preceded by custom adjective-pair selection and a pre-study assessing pair usefulness.
  • Dataset Resource: MT dimensions are Precision, Complexity, Grammaticality, and Transparency, whereas ATS dimensions are Linguistic Logic, Complexity, Clarity, and Predictability.Complexity is shared across tasks, while the dominant dimensions differ in their remaining indicators and reflect translation fidelity versus summarization cohesion.
  • Dataset Resource: The reduced adjective-pair sets showed strong cross-round alignment, with Spearman correlations around 0.8 for both text types.ATS showed no significant differences across experiments; MT differed only for Complexity, by 0.5 points on a 7-point scale, while retaining high correlation.
  • Dataset Resource: TextQ-ATS contains 91 samples and TextQ-MT contains 106, with each item rated by 10 to 20 annotators and averaged per item.The datasets were curated from evaluated text items and human QoE dimension ratings for developing QoE prediction models.
  • Dataset Resource: The datasets are publicly available under a CC BY-NC 4.0 license for non-commercial purposes.The resource can be accessed through the project’s GitHub repository.

4 Methodology

The methodology develops QoE prediction for German summarization and machine translation using human ratings at both fine-grained and overall levels. It compares transformer, linguistic-feature, hybrid, and multi-task approaches, with hybrid models combining contextual representations and explicit text properties.

  • Dataset and prediction targets: TextQ-German supports automatic QoE assessment from human ratings of generated German text, covering dimension-level and overall quality prediction.The datasets use scaled human ratings to train predictors for unseen samples.
  • Dataset and prediction targets: The experiments cover ATS dimensions of linguistic logic, complexity, clarity, and predictability, plus MT dimensions of precision, complexity, transparency, and grammaticality.Overall QoE is evaluated using subsets with overall ratings, including LLM-generated data.
  • Dataset and prediction targets: Prediction is formulated as regression on continuous item-level MOS obtained by averaging discrete 0–6 ratings across annotators.Models support both single-target prediction and simultaneous multi-target regression.
  • Model families: Because the datasets are limited, fine-tuned German transformer models provide an initial baseline for QoE prediction.Five pretrained German models are fine-tuned with output layers sized for single-target or four-target regression.
  • Model families: Linguistic-feature models compute 121 real-valued readability, lexical, syntactic, and morphological features, then select relevant subsets for prediction.Feature selection uses wrapper and embedded methods, including RFE, SFS, Lasso, and Elastic Net.
  • Model families: Hybrid models concatenate selected linguistic features with transformer representations, while multi-task learning predicts multiple QoE dimensions through shared representations.The hybrid language model is fine-tuned end-to-end; the hybrid SVM uses a two-stage procedure.

5 Results and Discussion

The results show that selected linguistic features provide complementary information to transformer representations, although performance varies by task, target, and data setting. Hybrid models improve dimension-level prediction in both tasks, while held-out validation confirms robust ATS and overall MT prediction but continued difficulty for fine-grained MT prediction.

  • Evaluation setup: RMSE is the primary model-selection metric, supplemented by MAE and R2 averaged across folds and dimensions.RMSE receives greater weight because larger deviations from human QoE ratings are particularly relevant.
  • Transformer and multi-label baselines: gbert-large yields the lowest RMSE for ATS, while gelectra-large performs best for MT; prediction errors are systematically higher for MT.The authors note that data quality or annotation reliability differences could partly explain the cross-dataset discrepancy.
  • Transformer and multi-label baselines: The multi-task model performs worst on both ATS and MT, though only slightly, possibly because limited training data restricts learning complex task interactions.The current multi-task implementation is essentially a regressor over language-model embeddings.
  • Linguistic feature selection: Selected linguistic feature subsets are notably small, and a linear regression model using them can rival fine-tuned transformer models.This result motivates combining hand-crafted features with neural representations.
  • Hybrid models: ATS hybrid SVM reduces RMSE from 0.8438 to 0.8254 and raises R2 from 0.2927 to 0.3664, while MT hybrid language modeling lowers RMSE from 0.9657 to 0.9127 and raises R2 from 0.4141 to 0.4720.The best hybrid architecture differs by task: SVM for multi-sentence summaries and neural hybrid modeling for mostly single-sentence translation.
  • Overall QoE and joint training: For overall QoE, the same backbones remain strongest; joint training improves MT but not ATS, and multi-task learning underperforms on both datasets.The best MT configuration uses the merged corpus, whereas the best ATS model trains on ATS alone.
  • Final validation: Held-out validation retains predictive capability for ATS dimension-level and overall QoE and overall MT QoE, while fine-grained MT dimension prediction remains challenging.Validation performance is generally lower but close to development results, supporting robustness of the dataset collection and QoE modeling.

6 Conclusion

TextQ-German provides a German QoE resource spanning machine translation and summarization, with validated perceptual dimensions and automatic prediction models. Hybrid models generally outperform transformer-only baselines, while held-out validation supports generalization for ATS and overall MT QoE, with fine-grained MT prediction remaining harder.

  • TextQ-German identifies four validated perceptual dimensions for each task: MT uses Precision, Complexity, Grammaticality, and Transparency, while ATS uses Linguistic Logic, Complexity, Clarity, and Predictability.
  • Hybrid QoE prediction models improve over pure transformer-based baselines in most experiments by combining neural embeddings with interpretable linguistic features.
  • Linguistic features selected through Sequential Feature Selection approach fine-tuned language-model performance while offering transparency, efficiency, and interpretability.
  • Held-out validation provides evidence of out-of-sample generalization for ATS and overall MT QoE, while fine-grained MT dimension prediction remains more challenging.
  • The study’s QoE operationalization focuses on text-observable attributes and omits contextual factors such as task utility, prior expectations, and emotional response.
  • Future work could extend the framework to additional NLG tasks, multilingual data, richer learning architectures, automatically discovered features, and uncertainty-aware evaluation metrics.

A Adjective Pairs

Tables 23 and 24 list the complete polar adjective pairs used in the experiments for machine translation and automatic text summarization.

  • Table 23 lists the complete polar adjective pairs for the machine-translation text type in German and English.
  • Table 24 lists the complete polar adjective pairs for the automatic-text-summarization text type in German and English.

B Generation Prompts

The appendix records the prompts and processing setup used to generate the LLM-based corpus extensions.

  • Each source text was inserted at the <source text> position and processed individually to generate the LLM-based corpus extensions.
  • OpenAI models were called through the Chat Completions API, while open-weight models were called through Ollama.

Machine translation (all models).

The prompt specifies English-to-German translation of each source text.

  • The machine-translation prompt instructs the model to translate the following English text to German.

Summarization, locally hosted open-weight models (except SauerkrautLM).

Die Vorgaben verlangen eine kurze deutsche Zusammenfassung des folgenden Textes. Sie soll prägnant sein, sich auf Hauptaussagen konzentrieren und Wiederholungen sowie Nebensächlichkeiten vermeiden.

  • Der folgende Text soll auf Deutsch kurz zusammengefasst werden.
  • Die Zusammenfassung soll drei bis fünf Sätze umfassen.
  • Sie soll Hauptaussagen und wichtigste Punkte hervorheben und Wiederholungen vermeiden.

Summarization, GPT-4o length-constrained variant (gpt-4o short).

Die Vorgabe fordert eine sehr kurze Zusammenfassung auf Deutsch. Sie soll maximal drei kurze Sätze mit möglichst weniger als 15 Wörtern pro Satz enthalten.

  • Der Text soll auf Deutsch sehr prägnant zusammengefasst werden.
  • Die Zusammenfassung darf höchstens drei kurze Sätze umfassen.
  • Jeder Satz soll möglichst weniger als 15 Wörter enthalten.

C Linguistic Features

Der Abschnitt dokumentiert verwendete linguistische Merkmale und die Leistung von Merkmalsauswahlmethoden für TextQ-ATS und TextQ-MT. Zusätzlich werden die Kennzeichnung bester RMSE-Werte und die Forschungsförderung angegeben.

  • C Linguistic Features: Tabelle 25 listet alle im Merkmalsauswahlprozess verwendeten Merkmale nach Merkmalstypen auf.
  • C Linguistic Features: Tabelle 26 zeigt die Leistung von Merkmalsauswahlmethoden auf TextQ-ATS nach Qualitätsdimension.
  • C Linguistic Features: Der beste RMSE-Wert je Qualitätsdimension ist fett hervorgehoben.Ein Gedankenstrich bedeutet, dass die Methode keine Merkmale auswählte und daher keinen gültigen RMSE erzeugte.
  • C Linguistic Features: Tabelle 27 zeigt die Leistung von Merkmalsauswahlmethoden auf TextQ-MT nach Qualitätsdimension.
  • C Linguistic Features: Die Forschung wurde durch die Deutsche Forschungsgemeinschaft im Projekt zur automatischen Abschätzung maschinell generierter Textqualität gefördert.Die Projektnummer lautet 436813723.
Loading 2608.18888v1…