Source-linked AI summary
EviSI: An Evaluation Agent for Simultaneous Interpreting
Ben Yan, Zongyao Li, Daimeng Wei, Weidong Liu, Huan Zhao, Chong Li, Yaode Wang, Yuzhe Shang
TL;DR
Simultaneous speech-to-speech interpreting must deliver target speech before the source ends, even when reformulation preserves meaning while diverging from written references. EviSI adapts MQM-inspired error analysis and deterministic scoring to assess semantic fidelity and oral expression, recovering strong aggregate system-ranking agreement with human judgments while individual-output agreement remains mixed.
Problem
Simultaneous speech-to-speech interpreting must deliver target speech before the source is complete, while valid reformulation can diverge from written references without being an error.
Method
EviSI combines shared source evidence, semantic-fidelity and oral-expression judgments, error reconciliation, and deterministic scoring inspired by MQM.
Results
EviSI recovers all 15 aggregate English-to-Chinese system preferences and reaches mean Kendall agreement of 0.707 for English to Chinese and 0.467 for Chinese to English, exceeding evaluated baselines.
Takeaways & Limitations
EviSI supports system-level comparison of simultaneous interpreting quality, while multilingual concordance without human ratings does not establish human validity.
Takeaways & Limitations
Individual-output agreement remains mixed, and textual assessment does not directly measure acoustic quality or listener comprehension.
Abstract
from arXiv · showhide
Simultaneous speech-to-speech translation requires understanding, translation and spoken delivery while the source stream continues. To support timely delivery and limit accumulated delay, systems adopt reformulation and summarization, which can preserve meaning while departing from written references. BLEU and COMET may not reliably distinguish such variation from semantic loss. We introduce EviSI, a large language model evaluation agent adapting the error analysis and penalty principles of Multidimensional Quality Metrics (MQM). It constructs shared source evidence, assesses semantic fidelity and oral expression, reconciles overlapping errors and scores deterministically. EviSI recovers the aggregate human system ranking for English to Chinese. Mean Kendall agreement with human system rankings within corpora reaches 0.707 for English to Chinese and 0.467 for Chinese to English, exceeding evaluated baselines. An extension across five directions shows positive concordance with COMET without human ratings. Individual output agreement with humans remains mixed.
1. INTRODUCTION
Simultaneous speech-to-speech translation must deliver target speech before the source stream ends, creating pressure for timely, coherent communication. Reformulation and summarization can preserve meaning while diverging from written references, so reference divergence is not necessarily an error.
- Simultaneous interpreting systems must generate and deliver target speech before the source is complete.
- Prolonged speech-to-speech output can delay subsequent information, motivating segmentation, reformulation and summarization.
- Interpreting strategies can preserve meaning while departing from written references.
- Fidelity and logical cohesion matter alongside audience expectations in interpreting evaluation.
- Reference divergence is therefore not necessarily an error.
2. METHOD
EviSI evaluates interpreting outputs through shared source evidence, specialized semantic and oral-expression judgments, reconciliation, and deterministic MQM-inspired deductions. Its tables define the evaluation comparisons and score presentation used in the experiments.
- 2.1. Evaluation workflow: EviSI coordinates specialized LLM roles in a fixed workflow, while code orchestrates, validates and scores without making semantic judgments.
- 2.1. Evaluation workflow: The source text is authoritative, while the auxiliary reference supports interpretation and alignment rather than wording.
- 2.2. Complementary assessment and reconciliation: Anchor, Event and Logic judgments assess factual identity, propositional coverage, and semantic relations, while valid reformulations incur no penalty.
- 2.2. Complementary assessment and reconciliation: Fluency evaluates natural, comprehensible oral expression rather than polished written style, and text cannot establish pronunciation, voice quality or prosody.
- 2.2. Complementary assessment and reconciliation: Semantic reconciliation compares Anchor, Event and Logic errors, then expression reconciliation compares retained semantic errors with Fluency errors.
- 2.2. Complementary assessment and reconciliation: Span overlap alone is not duplication, and retained and excluded reports remain inspectable.
- 2.3. Deterministic scoring: EviSI applies retained-error deductions without length normalization, following MQM’s penalty principle rather than producing a standard MQM score.
- 2.3. Deterministic scoring: Scores are deterministic conditional on fixed judgments, not necessarily across repeated LLM runs.
3. EXPERIMENTS
The experiments compare EviSI with established metrics on human system rankings and extend the comparison across multilingual corpora without human ratings. EviSI is strongest for aggregate EN→ZH ranking, but gains vary by corpus and individual-output agreement remains mixed.
- 3.1. Data and evaluation protocol: Five EN→ZH and four ZH→EN corpora provide human ratings for six commercial S2S interpreting systems, using their complete intersections without imputation.The evaluation contains 1,164 EN→ZH and 870 ZH→EN outputs from 194 and 145 sources, respectively.
- 3.1. Data and evaluation protocol: Baselines use Sentence BLEU, COMET20, COMET22 and COMETKiwi, with Kendall τb and Spearman ρ comparing six system means.System-level correlations weight corpora by source count, while within-corpus Kendall τb values are averaged equally.
- 3.2. Agreement with human system rankings: τb = 1.000 lets EviSI recover all 15 aggregate EN→ZH system preferences, exceeding Sentence BLEU’s 0.600 and the COMET family’s 0.200.EviSI ranks H2 first and H6 last; in ZH→EN, it matches 11 of 15 pairs and ties Sentence BLEU on τb = 0.467.
- 3.2. Agreement with human system rankings: 0.707 and 0.467 are EviSI’s mean system correlations for EN→ZH and ZH→EN, versus strongest baseline values of 0.413 and 0.333.Across all nine corpora, EviSI reaches ¯τb = 0.600 versus COMETKiwi’s 0.378, but obtains −0.067 on Z2 while COMET20 reaches 0.733.
- 3.3. Multilingual extension without human ratings: 0.733, 0.800 and 0.733 are the average positive concordances between EviSI and the three COMET metrics across five directions.The multilingual comparison uses four systems, M1 to M4, and corpus aggregates without human ratings; EN→JA is the weakest case, with disagreement over the top system.
- 3.4. Inspecting metric disagreements: 20.6 is EviSI’s percentile for the complete Z2 output, compared with 25.5 for humans and 73.4 for COMET20/22.The post hoc case motivates inspection of a semantic substitution but does not explain metric internals or override Z2’s negative correlation.
4. CONCLUSION
EviSI combines source-grounded evidence, specialized judgments, reconciliation and deterministic MQM-inspired deductions. It improves average agreement with human system rankings, while individual-output results and validation scope remain limited.
- Conclusion: EviSI combines shared source evidence, specialized judgments and reconciliation with deterministic deductions inspired by MQM.The approach is designed to make scoring inspectable while supporting adequate oral reformulation and semantic fidelity.
- Conclusion: EviSI improves average agreement with human system rankings, but individual-output results remain mixed.Multilingual concordance with metrics does not establish validity against human judgment.
- Conclusion: Independent tests, interannotator reliability, matched baselines, cost and repeatability measurements, and controlled validation remain necessary.The current rankings are coarse, historical metadata are incomplete, and the retrospective evaluation lacks an independent test set.
- Conclusion: Textual assessment does not directly measure acoustic quality or listener comprehension.The limitation bounds conclusions about spoken delivery beyond the evaluated text representations.