PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study

Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour

arXiv:2608.25574v1cs.CL

TL;DR

WER poorly captures semantic similarity, while the roles of encoder and decoder LLMs in ASR evaluation remain underexplored. The paper compares embedding metrics across models, layers, and pooling strategies and evaluates generative LLMs for hypothesis selection and qualitative error assessment. Both encoder and decoder approaches perform strongly when configured appropriately, with generative models providing accurate comparisons and interpretable qualitative feedback.

  • Problem

    WER poorly reflects semantic similarity, and the comparative roles of encoder- and decoder-based LLM representations in ASR evaluation remain underexplored.

  • Method

    The paper compares BERTScore and SemDist across encoder and decoder representations, layers, and pooling strategies, then evaluates generative LLMs for hypothesis selection and qualitative error classification.

  • Results

    Both BERTScore and SemDist achieve strong human agreement with suitable configurations; generative LLMs achieve very high agreement in pairwise selection and consistent error-severity ordering.

  • Takeaways & Limitations

    Encoder representations remain competitive and efficient, while generative models support accurate comparative judgments and more interpretable structured ASR feedback.

Abstract

from arXiv · show

Automatic Speech Recognition (ASR) is typically evaluated using Word Error Rate (WER), which poorly reflects semantic similarity. While embedding-based metrics correlate better with human judgments, the respective roles of encoder and decoder-based Large Language Models (LLMs) remain underexplored. This paper presents a comparative study of both families for ASR evaluation. We analyze BERTScore and SemDist across different LLMs, layers, and pooling strategies, showing that both metrics can achieve strong correlation with human judgments when properly configured. For decoder models, we investigate generative LLMs in two settings: pairwise hypothesis selection via prompting and direct qualitative error classification. Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.

I. INTRODUCTION

WER remains limited for ASR evaluation because surface-form matching can obscure perceived quality, motivating semantic metrics and comparison of encoder and decoder LLMs. The paper studies these representations alongside generative evaluation settings.

  • Motivation: WER’s strict matching rules can obscure meaningful improvements in perceived transcription quality.It is sensitive to casing and minor lexical differences.
  • Motivation: Semantic metrics using contextualized representations align more strongly with human judgments than surface-overlap metrics.Encoder architectures such as BERT have been especially influential in capturing contextual meaning.
  • Research gap: Decoder-only LLMs have received comparatively little attention for structured ASR evaluation despite their reasoning and generation capabilities.This motivates comparing encoder- and decoder-based representations.
  • Approach: The study examines sentence-level similarity representations, pooling strategies, pairwise hypothesis selection, and qualitative error assessment.These dimensions cover both embedding-based and generative evaluation approaches.
  • Approach: The evaluation uses HATS human judgments to compare encoder and decoder embeddings and assess generative models as direct evaluators.HATS provides detailed judgments of ASR outputs and supports comparison with human perception.
  • Motivation: Different metrics can rank ASR systems differently, so evaluation choices should reflect the intended application.This is especially relevant when transcripts are designed for human users.

B. Embedding-based Semantic Metrics

Embedding-based metrics aim to measure meaning preservation more faithfully than lexical error rates, but their interpretation and performance depend on representation choices. This paper extends the comparison to decoder-based LLMs and human-judgment benchmarks.

  • Semantic representations: Contextual embeddings capture sentence meaning beyond lexical overlap, making them suitable for evaluating ASR hypotheses.Recent semantic metrics use Transformer encoder representations such as BERT.
  • Metrics: BERTScore compares contextual token embeddings, whereas SemDist measures cosine distance between sentence embeddings.Both have correlated better with human judgments than WER, with performance depending on the underlying model and representation.
  • Interpretability: Embedding-based metrics remain difficult to interpret because cosine similarities lack an obvious semantic meaning.Prior work addresses this by incorporating semantic severity or analyzing errors by semantic impact.
  • Human evaluation: Human-judgment datasets enable direct comparison between automatic metrics and perceived transcription quality.Such benchmarks support evaluation against human perception rather than lexical agreement alone.
  • Human evaluation: HATS provides pairwise human preferences over French ASR hypotheses and reports stronger agreement for semantic metrics than WER.The benchmark confirms observations from proprietary datasets.
  • Research scope: The paper compares encoder and decoder LLM representations for semantic metrics and evaluates generative models as direct transcription-quality assessors.This addresses the limited prior focus on encoder-based semantic representations.

A. Evaluation Setup

The evaluation uses HATS human judgments and tests metric robustness across annotation-consensus levels. WER and CER provide lexical baselines for comparison.

  • Dataset: The study uses HATS as a reference for comparing automatic metrics against human judgments.The dataset supports evaluation under varying levels of annotator agreement.
  • Dataset: Three HATS subsets represent 100% agreement, ≥70% agreement, and the full dataset.These subsets vary the level of annotation ambiguity used to assess robustness.
  • Baselines: 63% and 77% agreement are obtained by WER and CER, respectively, on the 100% agreement subset.Embedding metrics produce continuous scores, so ties do not occur in these experiments.

B. BERTScore with Encoder and Decoder LLMs

BERTScore is evaluated across models and transformer layers against HATS human judgments, while SemDist is summarized across models and pooling strategies. Performance depends strongly on representation selection.

  • Evaluation: BERTScore correlations with HATS judgments are measured for embeddings from every transformer layer of each encoder and decoder LLM.This identifies representations that best capture semantic similarity perceived by annotators.
  • Layer selection: The optimal BERTScore layer depends strongly on the model, with intermediate layers often best and final layers optimal for some models.Layer selection is therefore an important hyperparameter for ASR evaluation.
  • Encoder models: Sentence-level fine-tuning mainly changes encoder models’ last layers, while earlier layers remain similar to corresponding CamemBERT representations.Its benefit is substantial for Sentence-CamemBERT-large but not systematic across model sizes.
  • Decoder models: Embedding-specialized Qwen3 models generally improve over their originals, with the largest differences in deeper layers.However, the last layers are not generally the most suitable BERTScore representations.
  • Overall comparison: Both encoder and decoder LLMs perform strongly with appropriate layers, with the best results from Sentence-CamemBERT-large and Qwen3-Embedding-8B.Encoder models remain comparable to the best decoder models despite substantially fewer parameters.
  • SemDist: SemDist agreement is reported across models and pooling strategies using last-layer representations.The table uses HATS annotations from the annotator-consensus subset.

C. SemDist with Encoder and Decoder LLMs

SemDist derives sentence-level similarity from pooled LLM token representations, and its performance depends strongly on model, layer, and pooling choices. Encoder and decoder models can both perform well, while encoders remain competitive in efficiency.

  • SemDist measures distance from cosine similarity between sentence embeddings formed by aggregating LLM token representations.
  • Model size alone does not reliably predict SemDist performance; layer and pooling choices can let smaller or mid-sized models match or exceed larger ones.
  • Mean and weighted pooling are generally strong, whereas last-token representations are usually suboptimal outside embedding-specialized models.Embedding-specialized Qwen models make last-token pooling more competitive, often matching or surpassing alternatives.
  • The optimal representation layer is model-dependent, with no universally best early, intermediate, or final layer.Layer selection is therefore a critical design choice for LLM-based semantic evaluation.
  • Encoder and decoder models can both achieve strong performance with suitable configurations.Sentence-CamemBERT-large is consistently strong with a compact architecture, while Qwen3-Embedding-8B reaches similar or slightly higher scores at substantially higher parameter cost.
  • The best overall results come from Sentence-CamemBERT-large and Qwen3-Embedding-8B, but intermediate-model rankings vary across pooling strategies and metrics.Sentence-CamemBERT-large benefits more from layer selection, whereas Qwen3-Embedding-8B is more stable across layers.

IV. LLM AS A JUDGE FOR ASR EVALUATION

The paper evaluates decoder LLMs’ native generative capabilities for ASR quality assessment rather than relying on vector representations. It studies pairwise hypothesis selection and interpretable labeling of individual hypotheses.

  • Decoder LLMs are prompted to reason directly about transcription quality instead of comparing embedding vectors.
  • The generative evaluation uses two settings: selecting the better of two hypotheses and assigning an interpretable quality label to one hypothesis.
  • LLM inferences are performed with the SDialog toolkit.

A. Pairwise Hypothesis Selection

The pairwise experiment tests whether generative LLMs can select the ASR hypothesis that best matches human preference. Prompted comparison supports high agreement and can consider multiple dimensions of transcription quality jointly.

  • A generative LLM selects, from two ASR hypotheses and a reference, the hypothesis that best matches human preference.
  • The one-shot prompt provides one annotated example, requests a brief justification, and ends with A or B for automatic prediction extraction.
  • 94% agreement is achieved by GPT-4.1 and 92% by open-weight Qwen3.5-35B on the total-consensus subset.
  • Model size alone does not determine performance: Qwen3-8B outperforms Gemma3-27B, and Qwen3.5-27B surpasses Qwen3-30B.
  • Generative LLMs outperform WER, CER, and the strongest previously evaluated embedding-based semantic metrics for this specific hypothesis-selection task.The models jointly consider lexical accuracy, semantic fidelity, grammatical coherence, and contextual information.
  • High agreement with human judgments suggests generative LLMs could reduce manual annotation needs when constructing perceptual evaluation datasets.

B. Direct Classification of Hypotheses

Direct classification asks an LLM to assess each reference–hypothesis pair independently using four qualitative error-severity labels. The labels provide interpretable structure, but their relationship with continuous semantic scores remains moderate.

  • Direct classification evaluates each reference–hypothesis pair independently to produce an absolute, interpretable quality assessment.
  • The four labels are identical, useful, bad, and incomprehensible, ranging from preserved meaning to complete loss of meaning.
  • Qualitative label distributions describe the nature and severity of transcription errors more explicitly than a single scalar metric.
  • Because HATS lacks qualitative labels, predicted categories are compared with SemDist from Sentence-CamemBERT-large using an independent semantic signal.
  • GPT-4.1 achieves the best overall correlations, while larger LLMs generally show stronger correlations with SemDist.
  • The four categories follow the expected semantic ordering, but substantial overlap—especially between useful and bad—explains the moderate correlations.
  • Qualitative judgments are not yet strong enough to replace continuous semantic metrics, although they provide meaningful descriptions and structured feedback.Improving consistency through prompting, fine-tuning, or hybrid strategies remains future work.

V. CONCLUSION

The paper evaluates encoder and decoder LLMs for ASR through semantic metrics, pairwise judgments, and qualitative error classification. Encoder representations provide efficient semantic metrics, while generative models support accurate comparisons and interpretable feedback.

  • The study examines LLM-based ASR evaluation from embedding metrics, pairwise hypothesis selection, and qualitative error classification.
  • BERTScore and SemDist can strongly agree with human judgments when model, layer, and pooling are appropriately selected.No single layer consistently dominates across architectures, and performance does not depend solely on model scale.
  • Encoder-based LLMs remain highly competitive with slightly better-performing decoder models while typically requiring lower computational cost.
  • State-of-the-art generative models achieve very high agreement with human preferences in pairwise hypothesis selection, outperforming WER, CER, and embedding-based semantic metrics.The task evaluates competing hypotheses as relative judgments of transcription quality.
  • LLM qualitative classification consistently orders error severity and supports interpretable distributions over error categories.This provides a more detailed view of system behavior than single-score metrics, although correlations with continuous semantic scores remain moderate.
  • Together, encoder representations and generative models offer complementary routes toward more human-aligned and interpretable ASR evaluation.

APPENDIX

The appendix documents the fixed instruction-style prompt used for pairwise ASR hypothesis selection. It presents a reference and two hypotheses, asks for reasoning, and requires a final choice marker.

  • The pairwise evaluation uses fixed instruction-style prompts without prompt engineering beyond an in-context example.The appendix reports the prompts used during inference, translated from French for readability.
  • Each prompt frames the reference as an exact transcription and supplies two incorrect hypotheses for comparison.
  • The model is asked to explain its reasoning and end with A, B, a, or b when undecided.
  • An example favors a hypothesis with a tolerated disfluency over one containing a more problematic grammatical error.
  • The template substitutes the input reference and two ASR hypotheses into corresponding fields.

B. Prompt for Direct Hypothesis Classification

The direct classification prompt asks an LLM to assign one ASR hypothesis to a four-level error category. It defines categories by semantic preservation and requires a JSON-only response.

  • The prompt casts the LLM as an expert ASR evaluator given a reference transcription and one hypothesis.
  • The classifier must assign exactly one of four categories: IDENTICAL, USEFUL, BAD, or INCOMPREHENSIBLE.
  • IDENTICAL covers exact matches or differences limited to case or hyphenation, while USEFUL preserves meaning despite minor errors.
  • BAD denotes partially altered meaning from significant errors, while INCOMPREHENSIBLE denotes completely lost meaning.
  • The response must contain only a valid JSON object conforming to the specified category schema.
  • The template inserts the reference and hypothesis used for each evaluation pair.
Loading 2608.25574v1…