PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Evaluate Speech Translation with Source-Aware Neural MT Metrics

Mauro Cettolo, Marco Gaido, Matteo Negri, Sara Papi, Luisa Bentivogli

arXiv:2511.03295v3cs.CLcs.AI

TL;DR

Source-aware MT metrics need textual sources, but ST often provides audio without reliable transcripts or source-reference alignment. This paper tests ASR and back-translation as text proxies and proposes cross-lingual re-segmentation; ASR is more reliable below a 20% WER threshold, while re-segmentation supports evaluation with negligible degradation.

  • Problem

    Whether automatically generated source text can preserve source-aware metric reliability in ST remains underexplored, particularly when transcripts are unavailable.

    Source: §1 · p. 3

  • Method

    The study compares ASR transcripts with reference back-translations as synthetic sources and proposes XLR-Segmenter to re-align source text with references.

    Source: §1 · p. 3

  • Results

    Below 20% case-insensitive WER, ASR is more reliable than back-translation, while XLR-Segmenter yields only negligible degradation without audio-text alignments.

    Source: §6 · p. 36

  • Takeaways & Limitations

    Synthetic source-aware metrics provide a reliable and effective way to evaluate ST systems, with practical source-selection recommendations based on operating conditions.

    Source: §6 · p. 37

    The low-resource experiments cover only Bemba-English, so they do not exhaust the challenges of low-resource conditions, especially pairs where both languages are low-resource.

    Source: §7 · p. 38

Abstract

from arXiv · show

Automatic evaluation of ST systems is typically performed by comparing translation hypotheses with one or more reference translations. While effective to some extent, this approach inherits the limitation of reference-based evaluation that ignores valuable information from the source input. In MT, recent progress has shown that neural metrics incorporating the source text achieve stronger correlation with human judgments. Extending this idea to ST, however, is not trivial because the source is audio rather than text, and reliable transcripts or alignments between source and references are often unavailable. In this work, we conduct the first systematic study of source-aware metrics for ST, with a particular focus on real-world operating conditions where source transcripts are not available. We explore two complementary strategies for generating textual proxies of the input audio, ASR transcripts, and back-translations of the reference translation, and introduce a novel two-step cross-lingual re-segmentation algorithm to address the alignment mismatch between synthetic sources and reference translations. Our experiments, carried out on two ST benchmarks covering 79 language pairs and six ST systems with diverse architectures and performance levels, show that ASR transcripts constitute a more reliable synthetic source than back-translations when word error rate is below 20%, while back-translations always represent a computationally cheaper but still effective alternative. The robustness of these findings is further confirmed by experiments on a low-resource language pair (Bemba-English) and by a direct validation against human quality judgments. Furthermore, our cross-lingual re-segmentation algorithm enables robust use of source-aware MT metrics in ST evaluation, paving the way toward more accurate and principled evaluation methodologies for speech translation.

1. Introduction

Source-aware neural MT metrics need text, but speech translation supplies audio and often lacks transcripts or aligned references. This paper studies synthetic text sources and cross-lingual re-segmentation for applying these metrics in ST.

1. Introduction

  • Reference-based evaluation compares hypotheses with human translations, while source-aware neural metrics also use source text; ST's audio input makes those metrics difficult to apply directly.Manual quality assessment is considered most reliable but costly and difficult to scale.Source: §1 · p. 2
  • The paper asks whether automatically generated source text can preserve source-aware metric reliability, which source-generation approach is preferable, and whether misaligned sources can be re-aligned without harming evaluation.The real-world setting often lacks manual transcripts, and long recordings may lack sentence-level audio-text alignments.Source: §1 · p. 3
  • The study evaluates two benchmarks covering 79 language pairs, six diverse ST systems, a Bemba-English pair, and synthetic metrics against human quality judgments.The setup also uses over 120,000 sentences and considers both cascaded and direct architectures.Source: §1 · p. 3
  • ASR transcripts are more reliable than back-translations when case-insensitive WER is below 20%, and XLR-Segmenter supports evaluation without audio-text alignments with negligible degradation.The study also reports that synthetic source-aware metrics correlate strongly with standard metrics and that their COMET and MetricX variants correlate more strongly with human judgments than their standard counterparts.Source: §1 · p. 4
  • The paper presents its solutions, controlled comparisons, realistic tests, low-resource experiments, human-judgment validation, and limitations across Sections 3–7.The main experiments separately assess synthetic sources, re-segmentation, and the combined approach in realistic conditions.Source: §1 · p. 4

2. Related Works

MT evaluation has moved from surface overlap toward learned metrics with stronger human-judgment correlations, while ST evaluation inherits MT metrics despite audio-specific source and segmentation challenges. This work investigates source-aware evaluation and re-segmentation strategies for ST.

2.1 Metrics for Machine Translation

  • Neural MT metrics such as COMET and MetricX use contextual representations and have shown stronger correlations with human judgments than traditional surface-based metrics.The literature also notes persistent challenges in reliability, robustness, generalizability, and comparing scores across studies.Source: §2.1 · p. 4

2.2 Metrics for Speech Translation

  • ST evaluation has received less metric-development attention than MT, and inherited approaches face segmentation mismatches and possible absence of gold source transcripts.Source-aware MT metrics can outperform end-to-end speech-based evaluation, but require high-quality transcripts; systematic study of source-reference misalignment in ST remains absent.Source: §2.2 · p. 5
  • The paper systematically assesses source-aware metrics and re-segmentation strategies for ST when transcripts or their alignment with references cannot be assumed.It builds on MT evaluation advances while addressing the speech modality's challenges to gold source text and alignment.Source: §2.2 · p. 5

3. Methodology

The methodology addresses missing source text with ASR or reference back-translation, then tackles source-reference segmentation mismatch through cross-lingual re-segmentation and boundary refinement.

3.1 Synthetic Source Generation

  • Synthetic source text can come from ASR transcription of input audio or back-translation of the reference, enabling source-aware MT metrics when ST benchmarks lack transcripts.The choice depends on scenario constraints such as model availability, computational budget, neutrality, source fidelity, and re-segmentation needs.Source: §3.1 · pp. 6–7
  • ASR models have narrower language coverage than multilingual MT models, and their performance varies across languages and conditions.ASR-derived sources may bias evaluation if the ASR system shares components or training data with the evaluated ST system; BT avoids this specific dependency, though residual bias may remain if the same MT model is used in opposite directions.Source: §3.1 · p. 6
  • Back-translation aligns synthetic sources with references by construction, whereas ASR sources require re-segmentation that adds processing costs and can introduce errors.ASR requires a speech-recognition system and often a resegmenter, while BT requires only an MT system and is typically less computationally demanding.Source: §3.1 · pp. 6–7

3.2 Source Re-Segmentation

  • XL-Segmenter aligns ASR source segments to target references indirectly by aligning them against a segment-wise back-translation of the target text.It applies the intra-lingual L-Segmenter to the ASR text and back-translated reference segments.Source: §3.2 · p. 7
  • XLR-Segmenter refines adjacent source-segment boundaries while keeping the target-segment split fixed, moving source words when the optimal split differs.The refinement addresses L-Segmenter's string-matching limitation by using embedding-based word alignments that capture semantic relationships.Source: §3.2 · pp. 7–8
  • XLR-SimAlign minimizes cross-alignments at candidate source splits, while XLR-LaBSE minimizes average cosine distance between source and target sentence embeddings.XLR-LaBSE computes 2N embeddings per input sentence, which can be parallelized on a GPU; the paper reports that XLR-SimAlign resolves typical XL-Segmenter boundary issues.Source: §3.2 · pp. 8–9

4. Experimental Setting

The experimental setup uses two multilingual speech-translation benchmarks, source-aware metrics, synthetic source generators, and diverse ST systems. It evaluates synthetic-source metrics against scores using manual transcripts through segment-level Pearson correlations.

4.1 Data

  • MuST-C covers English speech translated into 14 languages, while Europarl-ST covers nine European speech languages and 72 translation directions.The test sets contain 33,295 and 88,227 segments, respectively.Source: §4.1 · pp. 9–10

4.2 Metrics

  • The study uses COMET-22 and MetricX-24-Hybrid as source-aware metrics, alongside WER, BLEU, and LASER for ASR, translation, and segmentation analyses.COMET encodes source, hypothesis, and reference; MetricX-24-Hybrid combines reference-based and reference-free signals.Source: §4.2 · pp. 11–12

4.3 ASR and BT Models

  • Three ASR models generate transcripts, while MADLAD and NLLB back-translate references to create synthetic sources.OWSM is strong on English (WERos=12.70% vs. WERwh=11.33 and WERsm=11.24) but weak on Portuguese (39.35%), Romanian (27.69%), and Polish (27.35%); MADLAD and NLLB are substantially equivalent on MuST-C, but MADLAD is significantly better on Europarl-ST.Source: §4.3 · pp. 12–13

4.4 ST Models

  • The six ST systems comprise three direct systems and three cascades pairing each investigated ASR model with MADLAD; system coverage varies across benchmarks and language pairs.WhisperST cannot be used on MuST-C, which has only English speech, and OWSM-ST excludes the eight to-Polish tasks because it does not support translation into Polish.Source: §4.4 · pp. 13–14
  • Cascaded systems generally outperform direct systems; WhisperST is the only competitive direct system, while OWSM-ST performs particularly poorly on Europarl-ST.On MuST-C’s eight language pairs shared with Tsiamas et al. (2024), Whisper+MADLAD scores 31.8 BLEU, ranking fifth against a best average of 33.2.Source: §4.4 · p. 14

4.5 Metric Evaluation Criterion

  • For each benchmark, metric, synthetic-source method, and ST system, the study correlates segment-level scores against scores using manual transcripts with Pearson correlation.Correlations above 0.80 are treated as very strong; whole-test-set correlations between system rankings and scores are omitted from the main text because they exceed 0.99 for all methods and metrics.Source: §4.5 · p. 15

5. Experiments

These experiments evaluate synthetic sources for ST metrics by comparing ASR transcripts with back-translations under known alignments and automatic re-segmentation, including manual-transcript and realistic settings. They assess agreement with manual-source MetricX and how ASR’s relative advantage over back-translation depends on WER.

5.1 Comparison of Synthetic Sources with Known Audio-Reference Alignments

  • Both synthetic sources correlated strongly with standard source-aware metrics, and ASR generally outperformed BT when transcription quality was adequate.Correlations exceeded 0.80; percentage recovery of the correlation gap ranged from 67.63 to 97.07 for MetricX, while COMET remained highly correlated even with randomly shuffled sources. Potentially biased ASR conditions were excluded from later aggregate results.Source: Table 5 · p. 16, §5.1 · pp. 17–18, Table 6 · p. 18

5.2 Insights into Synthetic Source Comparison

  • ASR sources were preferable when WER was at most 20%, while BT won more often above that threshold; the choice showed no dependence on language pair or ST architecture.ASR won 87.4% of comparisons at WER≤20%, whereas BT won 85.5% above 20%; across all comparisons ASR won 78.6%.Source: §5.2 · pp. 19–20, Figure 2 · p. 20, Table 7 · p. 20

5.3 Source Re-segmentation of Manual Transcripts

  • On manual transcripts, XL-Segmenter’s degradation was limited to 1–2%, and boundary refinement improved it, with SimAlign slightly outperforming LaBSE and results robust to BT quality.The manually segmented scores served as an upper bound; minimal differences between manual and automatic segmentation showed that the segmenters could reconstruct the original segmentation.Source: §5.3 · pp. 22–23, Table 10 · p. 23

5.4 In-the-Wild Evaluation

  • WER improved by more than 7 absolute points with boundary refinement, while re-segmented transcripts approached manual-segmentation semantics in realistic conditions.For Whisper/NLLB on Europarl-ST, WERnp fell from 20.92 to 13.84. Whisper/MADLAD LASER scores were 0.8292/0.8308 on MuST-C and 0.8830 on Europarl-ST, only 2–3% below manual-transcript upper bounds; on MuST-C, re-segmentation even improved the comparison with manual audio segmentation (0.8292/0.8308 vs. 0.8256).Source: §5.4 · pp. 24–25, Table 11 · p. 25
  • After re-segmentation by XLR-SimAlign and XLR-LaBSE, the minimum WERnp degradation across the two corpora was 3.09 points on Europarl-ST (ASRwhspr−mdld, 10.40 to 13.49) and 4.64 points on MuST-C (ASRwhspr−mdld, 6.97 to 11.61).In the example, “in your city.” appears in the audio and source transcript but is omitted from the Italian reference translation and its BT; these errors do not alter the overall semantic content of the segments.Source: §5.4 · pp. 25–26
  • Refinement consistently lowers WER versus XL-Segmenter, and automatic-source alignment loses only 2–3% LASER compared with the manual-source upper bound.XLR-SimAlign has lower WER than XLR-LaBSE except for Farsi, where their values are 23.02 and 21.81, respectively.Source: §5.4 · p. 27
  • Re-segmented transcript WER has a quadratic relationship with BT WER, with R2 values from 0.868 to 0.905; high lexical divergence does not necessarily mean semantic divergence.For en-zh and en-fr, BT WER is 83.33 and 33.33, respectively, while their MetricX scores are 1.398 and 1.612, respectively.Source: §5.4 · p. 27
  • ASR is preferred in 61.2% of comparisons overall and 74.7% when WER is at most 20%, while BT is preferred in 85.5% when WER exceeds 20%.These comparisons cover language pairs and ST systems across both corpora.Source: Table 17 · p. 29, §5.4 · p. 31
  • ASR-derived correlations generally remain above 0.92 in realistic conditions; even high-WER exceptions remain above 0.80, with a minimum of 0.8140.The exceptions are SeamlessM4T transcripts on MuST-C and OWSM transcripts on Europarl-ST.Source: §5.4 · pp. 29–30
  • At least one ASR source outperforms every BT source in 9 of 11 cases: 5/5 ST systems on MuST-C and 4/6 on Europarl-ST.This supports ASR's greater effectiveness in the realistic setting, while the WER threshold distinguishes when BT is preferable.Source: §5.4 · p. 30
  • The owsmST/MuST-C averages are 0.9858 for r(whspr-mdld), 0.9754 for r(BTmdld), and 0.9615 for MetricX-ref.Synthetic MetricX variants outperform the source-free version across all language pairs despite BLEU scores ranging from 7.15 to 27.29.Source: Table 18 · p. 31, §5.4 · p. 31

5.5 Low-Resource Evaluation

  • The low-resource evaluation examines COMET and MetricX on Bemba-English, a language pair not explicitly covered by either metric.The supplied passage introduces this evaluation but does not report its findings.Source: §5.5 · p. 31

Evaluation on a Low-Resource Language Pair: Bemba-English

Bemba-English experiments confirm that synthetic sources can support source-aware ST evaluation in a low-resource setting, while direct human-judgment validation supports the synthetic metrics’ effectiveness. The results also show that reliability depends on both the metric and the quality of the synthetic source.

Evaluation on a Low-Resource Language Pair: Bemba-English

  • Bemba ASR models’ CER gap from the strongest model was smaller than their WER gap, indicating that word segmentation—not only acoustic fidelity—was a key difficulty.The study selected ASR transcripts from opposite performance ends to amplify the effect of transcription accuracy on metric behavior.Source: Evaluation on a Low-Resource Language… · pp. 32–33
  • Bemba-English results were consistent with the MuST-C and Europarl-ST findings, supporting the synthetic-source patterns in a low-resource setting.The authors describe this as evidence that the patterns are not an artifact of high-resource conditions.Source: Evaluation on a Low-Resource Language… · pp. 33–34
  • MetricX correlated best with its standard version when given reliable ASR; as ASR WER increased, correlation dropped sharply, while BT results fell between high- and low-quality ASR.COMET’s synthetic variants also correlated strongly with its standard version, but this appeared to reflect limited source-signal use.Source: Evaluation on a Low-Resource Language… · pp. 33–34

5.6 Validation against Human Judgments

  • ASRwhspr-based synthetic COMET and MetricX correlated with human Direct Assessment scores more strongly than their standard versions, while BT-based variants remained close.The comparison used 304 fully annotated items across five language pairs and three ST systems.Source: §5.6 · p. 35
  • Both ASRwhspr and BTmdld variants outperformed the QE metrics and the six reference-based metrics on human-judgment correlation, with COMET-BTmdld’s margin over COMET-ref negligible.COMET-BTmdld scored 0.5474 versus 0.5471 for COMET-ref.Source: §5.6 · p. 35

6. Discussion and Conclusions

The study finds that synthetic source text can make source-aware metrics usable for ST when manual transcripts are unavailable, with ASR preferred when its WER is at most 20% and BT a practical alternative. Cross-lingual re-segmentation supports evaluation without audio-text alignments, while the conclusions are reinforced by low-resource and human-judgment results.

6. Discussion and Conclusions

  • Both ASR and BT are effective substitutes for manual transcripts, but ASR is more reliable when case-insensitive, punctuation-free WER does not exceed 20%.Above that threshold, the authors report BT as the better alternative.Source: §6 · pp. 35–36
  • XLR-Segmenter enables reliable evaluation without audio-text alignments, with negligible degradation; its boundary-refinement step restores semantic correspondence between ASR and reference segments.The authors also report that COMET appears to use source text relatively little, although the broader observations hold for both COMET and MetricX.Source: §6 · p. 36
  • The findings extend to Bemba-English, and synthetic COMET and MetricX variants reach human-assessment correlations stronger than their standard counterparts.The standard metrics in turn outperform the other automatic metrics considered and their source-free variants.Source: §6 · p. 36
  • Applying the 20% ASR-WER rule requires estimating ASR error, but comparable operational performance is rarely known and a suitable test set is not guaranteed to be available.The authors specify case-insensitive WER without punctuation for the threshold.Source: §6 · p. 36
  • BT remains effective despite being derived from the reference: it retains source-originating information and its variants outperform reference-only counterparts on correlations with standard metrics and human judgments.The authors attribute this to residual semantic correspondence between BT and manual transcripts.Source: §6 · p. 37
  • BT offers broader language coverage, is generally cheaper to generate, and is more neutral, while ASR more closely approximates manual transcripts when unbiased and well aligned.ASR generation is more demanding, and ASR sources require re-segmentation to align with references.Source: §6 · p. 37

7. Limitations

The study’s scope is bounded by language coverage, validation scale, benchmark conditions, metric selection, and the quality of the systems used to generate synthetic sources.

7. Limitations

  • The text-proxy approach does not apply to languages without a standardized written form.Addressing these languages would require audio-direct multimodal source-aware metrics, which are outside this work’s scope.Source: §7 · p. 38
  • The main datasets cover high- and medium-resource languages, while Bemba–English is the only low-resource pair and cannot represent the full range of low-resource conditions.The study does not cover pairs where both languages are low-resource or cases without pretrained ASR or MT models.Source: §7 · p. 38
  • Human-judgment validation covers 304 items, five language pairs, and three ST systems, so its conclusions warrant caution.The authors say a more definitive assessment would require larger-scale annotation across more language pairs, systems, and domains.Source: §7 · p. 38
  • All benchmarks use clean, single-speaker recordings rather than speech recorded in natural conditions.The authors note these remain the only available ST benchmarks covering 79 language pairs.Source: §7 · p. 38
  • The analysis focuses on COMET and MetricX, two representative source-aware metric families, rather than a broader set of metrics.The authors report that MetricX is the only one significantly affected by the source, while findings with MetricX remain valid for COMET.Source: §7 · p. 38
  • Back-translation uses high-performing MT models, whereas the three ASR systems do not always reach comparable accuracy.The authors note that weaker MT models would likely reduce BT-source reliability; Bemba–English provides only single-pair evidence on weaker BT models.Source: §7 · pp. 38–39

Appendix A: On the Quality of ASR and MT Models

ASR quality varies substantially with speech conditions, while MT quality varies especially with training-data availability. The authors use this contrast to contextualize their synthetic sources and explain their model selection.

Appendix A: On the Quality of ASR and MT Models

  • ASR WER ranges from 1–3% on clean read speech to 20–50% on challenging meeting tasks and above 60% for severe dysarthria.The passage also reports WERs of 5–10% or higher on spontaneous speech and above 30% for English accents; clean-read results are cited to Gulati et al. (2020) and Baevski et al. (2020), and dysarthria results to Almadhor et al. (2023).Source: Appendix A · p. 39
  • MT performance varies by domain, language pair, and especially training-data availability, with substantial quality drops reported for very low-resource languages.High-resource pairs can achieve BLEU scores above 40 and COMET scores of 85–90, including linguistically distant pairs.Source: Appendix A · pp. 39–40
  • The authors include relatively poor ASR models but only good-quality MT models because ASR can be difficult even in common conditions, whereas MT’s main constraint is data availability.The appendix frames this choice as consistent with commonly available research systems.Source: Appendix A · p. 40

Appendix B: Cost of Generating Synthetic Sources

The appendix compares the resource demands of generating ASR- and BT-based synthetic sources and describes an accuracy–efficiency choice among ASR re-segmentation strategies.

Appendix B: Cost of Generating Synthetic Sources

  • BT-source generation is at least 2–8 times faster than ASR-source generation, depending on re-segmentation type and GPU batch-size parallelism.The comparison measures execution time excluding model loading and peak GPU memory on the es→it Europarl-ST test set, using a Tesla A40 GPU.Source: Appendix B · p. 40, Table B.1 · p. 40
  • The ASR pipeline requires four steps, while BT-source generation requires only the MT step.The authors identify the boundary-refinement Algorithm 1 as a major cost driver, especially with SimAlign; its sequential boundary updates prevent parallel processing.Source: Appendix B · p. 40
  • XL-Segmenter alone is the most efficient but least accurate option; refinement improves accuracy, with SimAlign favoring maximum accuracy and LaBSE lower latency.The authors describe LaBSE as preferable when only a minimal accuracy degradation is acceptable.Source: Appendix B · p. 41

Appendix C: The Two-Stage Re-Segmentation Algorithm at Work

The appendix illustrates how the two-stage algorithm re-segments a source transcript to match reference-translation segments, then refines segment boundaries using cross-lingual word alignments.

Appendix C: The Two-Stage Re-Segmentation Algorithm at Work

  • The controlled example starts with a reference transcript whose original segmentation is ignored, alongside segmented reference translations and their back-translations.The example illustrates the process rather than the long-recording setting’s full operating conditions.Source: Appendix C · p. 41, Figure C.1 · p. 41
  • L-Segmenter uses the transcript and back-translation as input, assigning WER labels to matches, substitutions, insertions, and deletions to re-segment the transcript.In the example, “Are not” are insertions near the end-of-segment marker and are initially assigned to the left segment.Source: Appendix C · pp. 41–42
  • The refinement stage uses SimAlign word alignments and places the moving end-of-segment tag where cross-alignments are minimized.In this example, zero cross-alignments correctly move “Are not” to the right segment.Source: Appendix C · p. 42

Appendix D: Systems’ Ranking Correlation

Synthetic-source COMET and MetricX scores preserve ST-system rankings almost identically to their standard versions across the reported settings. The sole noted exception is OWSM on MuST-C with known audio-reference alignment.

Appendix D: Systems’ Ranking Correlation

  • System-score correlations are above 0.99 in every reported case except OWSM on MuST-C with known audio-reference alignment.The table reports averages across language pairs for each test set and setting.Source: Appendix D · p. 42, Table D.1 · p. 42
  • The correlations compare each synthetic-source version of COMET or MetricX with its corresponding standard metric across six ST systems.The comparison covers both ASR- and BT-generated sources, and controlled and uncontrolled settings.Source: Appendix D · p. 42, Table D.1 · p. 42
  • The authors report that system rankings and differences between system scores are not significantly impacted by the chosen source or condition.Source: Appendix D · p. 42

Appendix E: COMET in the Wild (Realistic Conditions)

Under realistic conditions, COMET shows the same 20% WER threshold reported for MetricX when choosing between ASR and BT sources. The appendix also presents unbiased source-preference comparisons and ASR re-segmentation correlation tables.

Appendix E: COMET in the Wild (Realistic Conditions)

  • Tables E.1 and E.2 report per-language-pair averaged Pearson correlations for COMET using re-segmented ASR transcripts on MuST-C and Europarl-ST.The prose directs readers to compare these values with the corresponding BT-source values in Tables 5 and 6.Source: Appendix E · p. 43, Table E.1 · p. 43, Table E.2 · p. 44
  • For COMET, the 20% WER threshold observed for MetricX still distinguishes when ASR quality is sufficient to make ASR preferable to BT.The authors note that COMET depends only weakly on the source.Source: Appendix E · p. 43
  • ASR is preferred in 2,185 of 3,440 unbiased pairwise comparisons (63.5%), while BT is preferred in 1,255 (36.5%).Figure E.1 plots the cases by WER and MetricX scores for the ASR and BT sources, respectively; values were randomly changed by 1% to reduce overlap.Source: Figure E.1 · p. 45
  • Figure E.2 histograms show source-preference cases by transcription quality (WER) and back-translation quality (MetricX).They compare whether standard COMET correlates more highly with COMET using the ASR or BT source.Source: Figure E.2 · p. 45

Related papers

Loading 2511.03295v3…