PaperlayerView exact revision on arXiv

Source-linked AI summary

ETC-Bench: A Multilingual Benchmark for Evaluation-Side Transcript Canonicalization

Zhiheng Qian, Junhao Du, Jing Peng, Hanqi Li, Kunyang Peng, Feng Xue, Kai Yu

arXiv:2610.36527v1cs.CL

TL;DR

Speech does not always determine a unique written transcript, complicating ASR evaluation across transcript conventions. The paper defines ETC and builds a multilingual benchmark with a metric and taxonomy-guided agent skill; its agents outperform evaluated conventional systems on shared languages.

  • Problem

    Speech can leave alternative written forms unresolved, and WER may count those differences as errors, conflating model performance with transcript conventions.

    Source: §1 · p. 1

  • Method

    The paper defines ETC, constructs a multilingual minimal-pair benchmark and taxonomy, proposes SENA, and introduces TACS to guide LLM-agent canonicalization.

    Source: §1 · p. 2, §2.1 · p. 2, §2.3 · p. 3

  • Results

    TACS agents achieve higher SENA than evaluated conventional baselines on shared languages.

    Source: §1 · p. 2, §4.2 · p. 4

  • Takeaways & Limitations

    ETC-Bench provides a systematic testbed for evaluation-side transcript canonicalization across languages.

    Source: §5 · p. 4

    Extending ETC to speech and text translation requires task-specific definitions of admissible variation and semantic-preservation constraints, left for future work.

    Source: §1 · p. 2

Abstract

from arXiv · show

Written transcripts of the same speech can differ in ways that are not uniquely determined by the speech signal. We formulate evaluation-side transcript canonicalization (ETC), which maps multiple surface forms to a single canonical form when their differences are not determined by speech. To our knowledge, ETC-Bench is the first and most comprehensive multilingual benchmark for evaluation-side transcript canonicalization, covering 181 phenomenon types across 12 languages. Each instance is a minimal pair that shares a sentence context and differs only in a target span exhibiting one phenomenon. To evaluate both canonicalization success and semantic preservation, we propose Semantic-preserving Exact-match Normalization Accuracy (SENA), which combines exact agreement between independently canonicalized pair members with a pair-relative BERTScore constraint against information loss. We further introduce the Taxonomy-guided Agent Canonicalization Skill (TACS), which provides LLM agents with phenomenon descriptions and explicit canonicalization contracts. Experiments with five baseline configurations and three TACS-based agents show that the agents outperform the baseline systems on shared languages. ETC-Bench provides a systematic testbed for evaluation-side transcript canonicalization across languages.

1. INTRODUCTION

Speech can leave multiple written forms indistinguishable, causing WER to count transcript conventions as errors. The paper defines ETC and introduces a multilingual benchmark, a semantic-preserving metric, and a taxonomy-guided agent skill.

1. INTRODUCTION

  • WER can count speech-indistinguishable written forms, such as “2” and “two,” as errors, conflating ASR performance with transcript conventions and potentially biasing cross-lingual comparisons.The paper describes ETC as addressing variants whose differences cannot be resolved from speech.Source: §1 · p. 1
  • TN and ITN can serve as canonicalization subroutines, but their coverage is limited, and multilingual evaluation-oriented normalization remains limited despite existing practices.The cited practices include multi-reference evaluation, Sclite, and Whisper normalization; Whisper’s well-established normalizer is limited to English, and inappropriate behavior can damage semantic information in Indic scripts.Source: §1 · p. 1
  • ETC maps transcript variants to a consistent canonical form when their differences cannot be resolved from speech, while preserving the semantics encoded in speech.The canonicalizer processes a transcript using its language without access to another transcript of the same utterance.Source: §1 · p. 1
  • Extending ETC to speech and text translation requires task-specific definitions of admissible variation and semantic-preservation constraints, which the paper leaves for future work.Source: §1 · p. 2
  • The paper introduces ETC-Bench with 1,098 native-speaker-reviewed minimal pairs spanning 181 phenomenon types across 12 languages, proposes SENA, and releases TACS; TACS agents achieve higher SENA than evaluated conventional baselines on shared languages.SENA is named as Semantic-preserving Exact-match Normalization Accuracy.Source: §1 · p. 2

2. ETC-BENCH

ETC-Bench organizes transcript ambiguities into a multilingual benchmark of minimal pairs, then evaluates canonicalization using both exact agreement and semantic preservation. Its taxonomy distinguishes language-agnostic from language-specific phenomena.

2. ETC-BENCH

  • ETC-Bench contains 1,098 minimal pairs covering 181 canonicalization phenomena across 12 languages.The pairs share a sentence context and differ only in a target span instantiating the phenomenon.Source: §2 · p. 2, §2.2 · p. 2

2.1. Phenomenon Taxonomy

  • The five-category taxonomy separates language-agnostic phenomena from language-specific ones, with each phenomenon defined by a described pattern of transcript ambiguity.Numeric Expressions and Symbols and Markers are language-agnostic; Abbreviations, Orthographic Variants, and Homophonic Substitutions are language-specific.Source: §2.1 · p. 2

2.2. Benchmark Construction

  • Benchmark construction generated and filtered candidate pairs, sampled for phenomenon coverage, and used native-speaker review to produce the final 1,098-pair release.Twelve native speakers reviewed phenomenon definitions and sampled pairs; released counts reflect final curation, not the initial 100-pair-per-language sampling quota.Source: §2.2 · p. 2

2.3. Evaluation Metric

  • SENA measures the proportion of pairs whose independently canonicalized texts match exactly while their semantic-preservation score meets the pair-relative BERTScore constraint.The constraint is f_i + δ ≥ t_i, where t_i is original-pair similarity, f_i is the lower input-to-output similarity, and δ = 0.1.Source: §2.3 · p. 3

3. TACS: TAXONOMY-GUIDED AGENT CANONICALIZATION SKILL

TACS gives LLM agents taxonomy-based phenomenon descriptions and canonicalization contracts to guide consistent transcript canonicalization. Its instructions constrain replacements to local spans while preserving unrelated or unresolved text.

3. TACS: TAXONOMY-GUIDED AGENT CANONICALIZATION SKILL

  • TACS uses phenomenon descriptions and language-, category-, and phenomenon-specific contracts to guide canonical forms, including rewrites beyond conventional TN and ITN classes.This addresses the challenge that multilingual contextual knowledge alone does not ensure independently processed variants receive the same output.Source: §3 · p. 3
  • Agents identify phenomenon instances and resolve competing interpretations before making local replacements, preserving text outside selected spans and retaining unresolved or inapplicable spans.The instructions require non-overlapping replacements and do not prescribe separate model calls or intermediate outputs.Source: §3 · p. 3
  • TACS directs agents to process each record independently, without cross-record evidence; the skill contains no benchmark pairs or per-item targets.Loading begins with SKILL.md, which specifies batch I/O and global constraints.Source: §3 · p. 3

4. EXPERIMENTS

The experiments compare conventional normalizers with TACS-based agents on ETC-Bench using SENA. TACS agents outperform conventional systems wherever both have results, while their category-level strengths and remaining challenges differ.

4.1. Experimental Setup

  • Five conventional configurations and three TACS-based systems are evaluated with SENA; agent runs are repeated three times and their means reported.Each agent canonicalizes pair members in separate fresh sessions with independent workspaces.Source: §4.1 · p. 3

4.2. Results

  • TACS-GPT, TACS-DeepSeek, and TACS-Qwen reach macro-averages of 43.3, 43.8, and 68.8 across 12 languages; conventional systems score 3.7–9.4 on average over supported languages.Their language-wise scores exceed those of evaluated conventional systems wherever both have results.Source: §4.2 · pp. 3–4, Table 3 · p. 4
  • The four TN/ITN systems score 0.0–2.8 on orthographic variants and zero on homophonic substitutions, while Whisper’s English normalizer reaches 40.9 on English.In Bengali and Hindi, none of the pairs made identical by removing informative script marks passes the semantic-preservation condition.Source: Table 4 · p. 4, §4.2 · p. 4
  • TACS agents score highest on abbreviations and orthographic variants, while symbols and homophonic substitutions remain more difficult; TACS-Qwen leads all five categories.For TACS-Qwen, the semantic-preservation condition rejected none of 262 identical-output symbol/marker evaluations and 8 of 283 identical-output homophonic evaluations.Source: Table 4 · p. 4, §4.2 · p. 4
  • Table 5 shows TACS-DeepSeek unifying transcript variants left distinct by Whisper normalization in two real ASR-evaluation examples.The examples illustrate resolution of variants on real ASR transcripts.Source: Table 5 · p. 4, §4.2 · p. 4
  • Without TACS, GPT, DeepSeek, and Qwen score 3.5, 4.6, and 13.9, compared with 43.3, 43.8, and 68.8 with TACS.At δ = 0.10, the semantic-preservation condition rejects all 81 observed Whisper merges that remove informative Bengali and Hindi script marks.Source: §4.2 · p. 4

5. CONCLUSION

The conclusion defines ETC as an independent task and presents ETC-Bench and TACS as resources for studying canonicalization across languages. It reports limited conventional-normalizer coverage and notes that homophonic substitutions remain challenging.

5. CONCLUSION

  • ETC-Bench provides 1,098 minimal pairs across 181 phenomena and 12 languages, while TACS uses a taxonomy and canonicalization contracts to guide LLM agents toward broader coverage.The conclusion identifies homophonic substitutions as a continuing challenge and invites extensions to language coverage, phenomenon inventories, and sample diversity.Source: §5 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

The work is computational and releases a dataset; native-speaker reviewers provided linguistic quality control, but no human or animal subjects were studied and ethical approval was not required.

  • The work is a computational study that releases a new dataset.Source: §6 · p. 5
  • Native-speaker reviewers provided linguistic quality control but were not research subjects.Source: §6 · p. 5
  • No human or animal subjects were studied, and ethical approval was not required.Source: §6 · p. 5
Loading 2610.36527v1…