PaperlayerView exact revision on arXiv

Source-linked AI summary

Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

Anubhav Jangra, Adrian de Wynter

arXiv:2610.19166v1cs.CL

TL;DR

Metric capability is commonly assessed through correlations with human ratings, though those correlations can hide indicators of metric quality. The paper introduces a regression-based meta-evaluation and multilingual benchmark, finding that calibrated simple metrics can match strong LLM-judges and that ensemble gains are greatest on style.

  • Problem

    Automatic metrics are commonly evaluated by correlating their outputs with human ratings, but raw correlations can hide indicators of metric quality.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes regression-based meta-evaluation and tests it on a benchmark spanning 10 languages, four evaluation dimensions, and 32 metrics.

    Source: §1 · p. 2

  • Results

    Calibrated simple string and embedding metrics matched or exceeded strong LLM-judges on style; all-metric ensembles reached Ψstyle(ρall) = 0.78 versus Ψstyle(LaBSE or Verbatim) = 0.43 for the best single metric.

    Source: §1 · p. 2, §7.2 · p. 9

  • Takeaways & Limitations

    Metric assessment depends on human-label skews and annotator agreement, so raw correlations alone can mask differences in metric quality.

    Source: §1 · p. 2, §8 · p. 10

    Pairwise ground truth was limited to a single higher-order generation preference, represented by differences in ordinal ratings.

    Source: §5.2 · p. 5

Abstract

from arXiv · show

Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG) benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents.

1 INTRODUCTION

Correlation-based evaluation can misrepresent metric utility for subjective judgments and is difficult to compare across settings. The paper proposes calibrated regression-based meta-evaluation and applies it to a multilingual style-personalized generation benchmark, finding that simpler metrics can match strong LLM-judges after calibration.

1 INTRODUCTION

  • Correlation can misestimate metric utility because it assumes a consistent score relationship, is sensitive to ties and label skews, and lacks a human-agreement reference point.These issues make correlation-based estimates difficult to compare across tasks, annotators, and datasets.Source: §1 · p. 1
  • The proposed framework calibrates metric outputs to human ratings, predicts human consensus with a regressor, and anchors performance between a constant predictor and averaged human agreement.This produces a quality estimate intended to be comparable across evaluation settings.Source: §1 · p. 1
  • The benchmark covers 10 typologically diverse languages, four evaluation dimensions, and 32 metrics across string-based, embedding, and LLM-judge paradigms, including ensembles.The study examines how ensembles compare with their constituent metrics.Source: §1 · p. 2
  • Calibrated string- and embedding-based metrics captured style at least as well as the strongest LLM-judges, despite lower raw correlations with human ratings.The study also found no single metric performed uniformly well across languages or dimensions.Source: §1 · p. 2

2 RELATED WORK

Prior work documents instability and limitations in correlation-based meta-evaluation, while multilingual metric evaluation remains limited beyond English. This paper examines multiple metric paradigms across languages and subjective dimensions using regression-based meta-evaluation.

2 RELATED WORK

  • Correlation-based meta-evaluation can be inconsistent across datasets, sensitive to outliers and ties, and difficult to interpret when annotator disagreement reflects genuine subjectivity.Prior analyses also report that metrics behave differently across datasets and systems.Source: §2 · p. 2
  • Style-personalized text generation is a subjective rewriting task assessed here for style accuracy, factual accuracy, leakage, and usability.These dimensions cover stylistic alignment, adherence to the source, reference-derived content, and usefulness as a rewrite.Source: §2 · pp. 2–3
  • Multilingual benchmarks have often focused on general response quality or preference and evaluated LLM-judges alone using correlations or pairwise accuracy.Prior work reports lower LLM-judge agreement in low-resource and non-Latin-script languages.Source: §2 · p. 3

3 PROBLEM DEFINITION

The paper frames style-personalized generation as rewriting source content in a reference style, with human judgments across ordinal dimensions. Its meta-evaluation goal is to score how well metrics capture those human assessments comparably across metrics and languages.

3 PROBLEM DEFINITION

  • A style-personalized generator rewrites an origin document in the style characterized by reference exemplars.The generator maps an origin and reference to a candidate generation.Source: §3 · p. 3
  • Each corpus instance pairs candidate generator outputs with multiple human ordinal judgments on evaluation dimensions.The judgments are provided by annotators for generated outputs.Source: §3 · p. 3
  • The meta-evaluation functional Ψ should score how well a metric captures a human evaluation dimension and make scores comparable across metrics and languages.The formulation also considers how much of a human construct one metric recovers that another misses.Source: §3 · p. 3

4 MULTI-LINGUAL SPTG DATASET CURATION

The benchmark was built from English source and reference texts, translated and corrected for multilingual use, then populated with generated paraphrases and human judgments. Its final corpus spans 10 languages and four dimensions, with annotation distributions and agreement varying by dimension.

4.1 FROM ENGLISH CORPUS TO MULTI-LINGUAL CORPORA

  • The English starting corpus contained 1,002 origin-reference pairs drawn from long-form text sources.References came from Reddit short stories and Amazon product reviews; origins covered several text domains.Source: §4.1 · p. 3
  • Texts were machine-translated into 36 target locales, after which native annotators edited most items, typically 80–99% of the 1,502 items per locale.Source: §4.1 · p. 4

4.2 STYLE PERSONALIZED TEXT FOR INDIVIDUAL LANGUAGE CORPUS

  • 10,038 instances across 10 languages each include two candidate paraphrases, three-annotator ratings on four dimensions, and a pairwise preference.Average inter-annotator agreement was higher for leakage (0.85) and factual accuracy (0.79) than for style (0.76) and usability (0.75).Source: §4.2 · p. 4
  • Candidate paraphrases were generated from localized source and reference texts, and a nine-model pool was inspected to select coherent outputs.The annotation study rated paraphrases on four ordinal dimensions and collected pairwise preferences; the detailed study covered 10 languages.Source: §4.2 · p. 4

5 PROPOSED METHODOLOGY

The methodology combines statistical diagnostics with a regression-based meta-metric to assess how well metric outputs predict human judgments. Its normalized score compares metric performance with a constant-mean baseline and annotator agreement.

5.1 STATISTICAL DIAGNOSTICS

  • Statistical diagnostics measure pointwise rank alignment with Spearman’s ρ and pairwise preference alignment with Cohen’s κ.Pairwise evaluation uses differences in average ordinal ratings and excludes non-decisive instances.Source: §5.1 · p. 5
  • Relative-agreement diagnostics estimate a metric’s unique contribution as the incremental R2 it adds beyond another metric when predicting ranked human labels.The remaining joint-model variance captures shared contribution; this analysis is limited to pointwise metrics scored on the same per-item units as human labels.Source: §5.1 · p. 5

5.2 META-METRIC: PREDICTING HUMAN LABELS FROM METRIC OUTPUTS

  • The meta-metric calibrates metric scores to the human-label scale, then trains a regressor to predict reliability-weighted human consensus.It evaluates held-out predictions against a constant-mean baseline and annotator agreement, making scores comparable across language-dimension settings.Source: §5.2 · pp. 5–6
  • Metric quality is the fraction of the gap between the mean-prediction baseline and annotator error that the metric closes.A score of 0 adds nothing beyond the baseline, while 1 matches a single annotator’s agreement with consensus; values above 1 are possible when an annotator is noisier than consensus.Source: §5.2 · p. 6

6 EXPERIMENTAL SETTINGS

The experiments compare reference-based metrics with LLM judges and an ensemble across four evaluation dimensions. They use corresponding style or content references for reference-based metrics, while judges receive both references and the generated text.

6.1 OFF-THE-SHELF EVALUATION METRICS

  • Reference-based metrics use the style reference for stylistic accuracy and leakage, and the content source for factual accuracy; LLM judges receive both references and the generated output.The ensemble combines metrics across paradigms to assess compositional improvements.Source: §6.1 · p. 6
  • String-based measures include edit distance, character 3-gram similarity, overlap, length ratio, punctuation-rate difference, and token-entropy difference.These measures are described as inexpensive, language-agnostic, and applicable to all ten languages.Source: §6.1 · p. 6
  • Embedding metrics measure semantic or style distance between generated outputs and references using cosine distance.The study uses LaBSE for semantic embeddings and several language-specific or multilingual style representations.Source: §6.1 · p. 6
  • Table 2 reports pointwise Spearman’s ρ and pairwise Cohen’s κ on the test split, averaged across human annotations from ten languages.It includes selected best-performing metrics; results for all metrics are reported in Table 6.Source: Table 2 · p. 7

7 FINDINGS

Calibrated meta-evaluation showed that simple metrics can match or outperform strong LLM-judges on style, while ensembles helped most where style signal was spread across metrics. Metric performance also depended on annotation properties, and diagnostic findings varied by evaluation dimension.

7.1 STATISTICAL DIAGNOSTICS OF EVALUATION METRICS

  • Raw pointwise correlations were modest overall: the best metric reached 0.56, with leakage highest at ρ=0.76 and stylistic accuracy lowest at ρ=0.40.LLM-judges had the strongest overall alignment, while LaBSE was the strongest reference-based metric at 0.35.Source: §7.1 · p. 7
  • Pairwise agreement exceeded pointwise alignment, but the comparison was not fair because pairwise evaluation covered only instances with a distinctly superior paraphrase.Strong LLM-judges approached the human ceiling on leakage and factual accuracy (κ ≥0.90), while stylistic accuracy remained hardest (κ ≤0.52).Source: §7.1 · p. 7
  • Only LLM-judge metrics added net positive variance across all dimensions; proprietary judges added +0.22 to 0.37 in leakage, while the reverse added ∼+0.02.LaBSE was a non-LLM exception, adding +0.20 over open-weight judges in leakage; stylistic accuracy showed near-collapse in unique information across metrics.Source: Figure 1 · p. 8, §7.1 · p. 8

7.2 META-EVALUATION OF METRICS

  • After calibration, even the strongest LLMs underperformed simpler string-based metrics and LaBSE on style accuracy.Reference-based metrics also performed competitively with the best LLM-judges on factual and leakage dimensions.Source: §7.2 · p. 8, Figure 2 · p. 9
  • Ψstyle(ρall) = 0.78, versus Ψstyle(LaBSE or Verbatim) = 0.43; ensembles gained most on style accuracy.The authors attribute this to style signal being spread thinly across multiple metrics, unlike content-based dimensions where signal is concentrated in a few metrics.Source: Figure 2 · p. 9, §7.2 · p. 9
  • Metric performance across languages depended on label spread and annotator agreement; English scored worst on all four dimensions, combining highest agreement with least label spread.Bengali, Korean, French, and English labels favored the best rating for 84-94% of candidates, while Japanese usability exceeded one because one annotator disagreed strongly with the other two.Source: §7.2 · p. 9

8 CONCLUSION

The conclusion summarizes a regression-based approach to evaluating metrics for subjective text generation and a multilingual benchmark study. It reports that metric performance varies with human-label properties and that ensembles improve most on style, where signal is spread across metrics.

8 CONCLUSION

  • The paper proposes regression-based meta-evaluation to assess metrics on subjective tasks such as style evaluation.It applies the framework to a multilingual style-personalized text-generation dataset annotated across 10 languages and four dimensions.Source: §8 · p. 10
  • Metric assessment depended on human-label skew and annotator agreement, and the best-performing metrics varied across languages and evaluation dimensions.The conclusion argues against uncalibrated metric-human correlations because raw correlations masked indicators of metric quality.Source: §8 · p. 10
  • Ensembles improved most on style, where signal was spread across many metrics; the conclusion favors learned metric combinations as a more faithful measure of capability.Source: §8 · p. 10

A SELECTING THE REGRESSOR FOR META-EVALUATION

The section compares 16 regression models for meta-evaluation and selects the best-ranking model. It also documents the multilingual corpus used to build the benchmark.

A SELECTING THE REGRESSOR FOR META-EVALUATION

  • The comparison covered 16 regressors from linear, kernel and instance-based, tree-ensemble, and neural-model families, trained on the same calibrated outputs and folds.The test split contained roughly 100 instances.Source: A SELECTING THE REGRESSOR FOR… · p. 15
  • 2.4 and 2.9 were the lowest mean ranks across languages and dimensions, achieved by ExtraTrees and RandomForest, respectively.The study selected ExtraTrees with 300 trees and minimum leaf size 3 for meta-evaluation.Source: Figure 3 · p. 15, A SELECTING THE REGRESSOR FOR… · p. 15
  • The English corpus paired style references, mostly creative short-form writing with some Amazon reviews, with origins spanning fiction, Wikipedia, IMDB, product reviews, and news.The table describes 1,002 annotated instances drawn from 1,502 translated source instances.Source: Table 3 · p. 16

B DATASET CURATION (ADDITIONAL DETAILS)

The dataset curation details describe translation review, the resulting text versions, the extent and types of edits, and annotator reliability weighting.

B DATASET CURATION (ADDITIONAL DETAILS)

  • Native-speaking annotators reviewed raw translations for accuracy, fluency, and locale appropriateness, producing human-corrected versions alongside the English source and machine translation.Each instance consequently had three text versions for a language.Source: B DATASET CURATION (ADDITIONAL DETAILS) · p. 16
  • Character-level edits ranged from under 1% in five locales to 33.9% for French, 23.2% for German, and about 22% for Zulu and Vietnamese.In French, Vietnamese, and Turkish, 97–99% of items were edited; annotators also left 5,461 free-text notes.Source: B DATASET CURATION (ADDITIONAL DETAILS) · p. 16
  • Edit notes were grouped into recurring correction types including fluency, lexical, grammatical, spelling, mistranslation, and dialect-specific issues.The notes were categorized by recurring types, with further examples referenced in Table 5.Source: B DATASET CURATION (ADDITIONAL DETAILS) · p. 16
  • Annotator reliability was calculated as leave-one-out Spearman correlation and used as the weight for each annotator in the meta-metric.Each annotator’s ratings were compared with the mean rating of the other two.Source: B DATASET CURATION (ADDITIONAL DETAILS) · p. 16

C EXTENDED RESULTS

The extended results collect pointwise and pairwise metric diagnostics, per-language comparisons, metric-information decompositions, ensemble contributions, and LLM-judge parsing errors.

C EXTENDED RESULTS

  • Table 6 reports pointwise Spearman ρ and pairwise Cohen’s κ for metrics across four dimensions and their aggregate, averaged over human annotations from 10 languages.Pairwise alignment excludes instances where the two paraphrases have equal ratings.Source: C EXTENDED RESULTS · p. 16
  • The appendix reports parsing-error distributions for all LLM-judge outputs generated for pointwise and pairwise metrics, alongside aggregate meta-evaluation results over all 10 languages and extensive per-language results.Source: C EXTENDED RESULTS · p. 16
  • Figure 5 compares per-language pointwise Spearman ρ for the top two overall metrics in each paradigm and evaluation dimension.Figure 4 presents per-language leave-one-out reliability correlations for the three annotators.Source: Figure 5 · p. 17, Figure 4 · p. 17
  • Figures 6–8 decompose metric information using ∆R2: across metrics, between paradigms, and for metric pairs; brighter Figure 8 cells indicate more relative information.Comparisons are blank where a paradigm has no applicable metric or dimension.Source: Figure 6 · p. 18, Figure 7 · p. 18, C EXTENDED RESULTS · p. 18
  • Figure 9 shows each constituent metric’s normalized share of ensemble out-of-fold permutation importance across 40 language-dimension combinations.Importance is based on the increase in out-of-fold MAE; negative values are clipped to zero before normalization, and unavailable metrics are marked gray.Source: C EXTENDED RESULTS · p. 16, Figure 9 · p. 20

D PROMPTS

The appendix provides prompts used in dataset construction and evaluation. They cover English-to-language translation, style-personalized generation, and pointwise and pairwise LLM judging.

D PROMPTS

Loading 2610.19166v1…