PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubric-Aligned Disentangled Evaluation of Human Simultaneous Interpreting

Ziyu Zhang, Satoshi Nakamura

arXiv:2609.11131v1cs.CL

TL;DR

Automatic segment-level SI evaluation lacks rubric-aligned metrics that preserve separate meaning, delivery, and synchrony dimensions. The paper builds a professionally annotated corpus and a dual-head LoRA-adapted COMET-KIWI evaluator, which improves held-out correlations over frozen COMET-KIWI while targeting stable formative-assessment rankings. Because the model is text-only and human agreement is limited, the results are interpreted relative to human consistency and do not cover multimodal latency modeling.

  • Problem

    Existing SI assessment frameworks distinguish meaning transfer, delivery quality, and synchrony, but lack scalable automatic segment-level evaluation aligned with these rubrics.

  • Method

    The paper constructs a professionally annotated 1,101-segment corpus and adapts COMET-KIWI with LoRA and separate regression heads for LQ and EXP.

  • Results

    Pearson correlations reach 0.388 for LQ and 0.301 for EXP on the held-out test set, improving over frozen COMET-KIWI.

  • Takeaways & Limitations

    The framework targets stable rubric-aligned ranking signals for scalable formative SI feedback rather than absolute agreement with a single rater.

  • Takeaways & Limitations

    The evaluation is text-only, uses a moderate dataset, reflects rater scale variability, and focuses on Chinese–English SI, leaving multimodal latency modeling for future work.

Abstract

from arXiv · show

Human simultaneous interpreting (SI) is commonly assessed with analytic rubrics separating meaning transfer, delivery quality, and temporal synchrony, yet no automatic metric is designed for rubric-aligned segment-level SI evaluation. We construct a professionally annotated corpus of 1,101 SI segments with scores for meaning transfer (LQ), delivery quality (EXP), and perceived latency (LAT). We show that structured LLM prompting and scalar supervision collapse rubric dimensions, yielding near-zero correlation with human ratings and strong cross-dimension coupling. To isolate supervision structure under identical backbone capacity, we introduce dual regression heads on a LoRA-adapted COMET-KIWI encoder. On a held-out talk-level test set, the model achieves Pearson correlations of 0.388 (LQ) and 0.301 (EXP), improving over frozen COMET-KIWI. Given low absolute rater agreement, we interpret results relative to human consistency and target stable ranking signals for formative assessment.

1. Introduction

Human simultaneous interpreting requires separate assessment of meaning transfer, delivery quality, and temporal synchrony, but existing scalable metrics do not preserve these rubric dimensions. The paper therefore develops a rubric-aligned framework using structured supervision and disentangled neural prediction.

  • Motivation: Segment-level SI evaluation is needed because professional rubrics distinguish meaning transfer, delivery quality, and synchrony, whereas existing frameworks lack scalable fine-grained diagnostics.The paper operationalizes these dimensions as LQ, EXP, and LAT.
  • Motivation: Written-translation metrics are poorly matched to SI’s incremental processing, temporal constraints, reformulation, and omission, producing discrepancies with human interpreting judgments.These limitations motivate interpreting-specific evaluation rather than direct reliance on conventional MT metrics.
  • Prior automatic evaluation: LLM rubric prompting yields strong cross-dimensional coupling, with dev-set correlation approximately 0.90 despite explicit scoring-order instructions.The result suggests that instruction-level control does not preserve rubric structure.
  • Research question: The paper hypothesizes that scalar supervision collapses ranking structure when analytic dimensions are correlated and rater scales vary.It tests whether parameter-level structural separation is needed to preserve rubric-aligned signals.
  • Contribution: The proposed framework combines a 1,101-segment professionally annotated corpus with LoRA-adapted COMET-KIWI and disentangled regression heads for meaning transfer and delivery quality.The stated goal is stable ranking signals for formative assessment rather than certification replacement.

2. Related Work

Prior SI assessment separates rubric dimensions, but automatic evaluation remains limited by written-translation metrics and the lack of rubric-supervised neural models. Recent LLM evaluators offer instruction-level control but raise concerns about preserving rubric distinctions.

  • MT evaluation metrics: Most automatic metrics were designed for written machine translation, using surface similarity or pretrained encoders to approximate scalar human quality judgments.The cited metric families include BLEU, TER, METEOR, chrF, BERTScore, BLEURT, YiSi, COMET, and COMETKIWI.
  • SI-specific challenges: SI’s incremental real-time production and latency–quality trade-offs create discrepancies between MT-oriented scores and human judgments under summarization and reformulation.Ear–Voice Span is one classical objective latency measure, but perceived SI quality involves interpreting-specific phenomena.
  • Rubric-based SI evaluation: Professional and academic SI frameworks separate meaning transfer and delivery quality, yet prior work lacks neural evaluators explicitly trained to disentangle these dimensions under rubric supervision.This leaves a gap between structured assessment principles and automatic segment-level modeling.
  • LLM evaluators: LLM-based SI evaluation relies mainly on instruction-level control, motivating concerns about whether prompting can reliably preserve rubric-level distinctions.The paper studies this issue through structured zero-shot and few-shot scoring strategies.

3. Dataset and Rubric

The corpus combines BSTC-derived and newly collected consented SI recordings, uses professional double ratings across rubric dimensions, and enforces talk-level data separation. Perceived latency is annotated separately from objective delay and shows weak alignment with that delay.

  • Dataset and Split: The corpus combines BSTC-derived segments with newly collected licensed TED-style recordings, normalized transcripts, and talk-level splitting to prevent discourse leakage.Sources include official transcripts where available and ASR transcripts for collected recordings.
  • Dataset and Split: Professional raters double-rate each segment under the analytic rubric, with individual ratings retained as separate supervision instances.The split contains 839 training, 87 development, and 169 test segments across 48, 5, and 8 talks, respectively.
  • Dataset and Split: The data cover English→Chinese and Chinese→English directions after semantic segmentation, transcript normalization, and removal of incomplete or unavailable segments.These preprocessing steps precede the talk-level split.
  • From Professional Framework to Segment-Level Operationalization: The rubric converts professional analytic scoring into segment-level neural supervision for meaning transfer, delivery quality, and perceived latency.A deterministic gating rule sets EXP to zero when LQ indicates severe meaning failure.
  • Latency Dimension Analysis: LAT is scored on a 0–3 scale alongside objective segment-level onset delay.The table caption identifies the segment-level rubric summary and its 0–3 range.
  • Latency Dimension Analysis: Perceived latency has mean 2.13 and standard deviation 0.79, but correlates weakly with objective delay: Pearson -0.048 and Spearman -0.034.LAT also correlates with LQ at 0.38 and EXP at 0.43, indicating a latency–quality trade-off.

4. Method

The method models segment-level LQ and EXP prediction with a LoRA-adapted COMET-KIWI encoder and independent regression heads. Residual prediction and variance regularization address noisy supervision, while the text-only design leaves LAT and acoustic delivery cues for future work.

  • Model formulation: The model predicts LQ and EXP from source–interpretation text using a pretrained COMET-KIWI encoder and multi-output regression.EXP is treated as a text-based lower bound because prosody, pauses, and fluency are not directly modeled; LAT is left for future multimodal work.
  • Design assumptions: The method assumes rubric dimensions are analytically separable, individual ratings capture scale variability, and formative assessment primarily needs stable rankings.These assumptions define the intended supervision and evaluation target.
  • Residual Prediction and Objective: Residual prediction represents each dimension as a mean score plus a learned deviation and combines LQ and EXP mean-squared errors with a variance regularizer.The displayed objective is the training loss associated with these terms.
  • Optimization: Training uses a heads-only first epoch followed by joint head-and-LoRA updates, with the best checkpoint selected by the summed LQ and EXP Pearson correlations.LoRA targets attention Q and V projections with r=8, α=16, and dropout 0.1.

5. Experiments

Experiments use talk-level splits, rubric-specific baselines, correlation-based evaluation, and inter-rater analyses to assess ranking alignment under subjective segment-level scoring.

  • Experimental setup: Talk-level splits evaluate generalization to unseen talks using Pearson correlation as the primary ranking-alignment measure.Spearman correlation, prediction variance, cross-dimension correlation, MSE, significance tests, and bootstrap resampling provide complementary analyses.
  • Baselines: The baselines isolate supervision structure through frozen and fine-tuned COMET-KIWI variants, scalar and dual heads, and structured prompt-based LLM evaluation.The comparison includes frozen representations, frozen-encoder dual heads, single-head LoRA fine-tuning, and prompt-based evaluation.
  • Human reliability: Inter-rater reliability is estimated on overlapping dual-annotated segments using ICC and pairwise Pearson correlations.The analysis includes 367 LQ segments and 345 EXP segments, with ICC measuring absolute agreement or consistency after scale shifts.
  • Human reliability: Mean pairwise human correlations are 0.264 for LQ, 0.286 for EXP, and 0.223 for LAT, establishing a subjective human–human reference ceiling.Absolute agreement is low, whereas consistency-level reliability is moderate.

6. Results

Prompting and scalar supervision fail to preserve rubric dimensions, whereas the dual-head model improves test-set alignment and reproduces human-like cross-dimensional dependence.

  • 6.1. Prompt-Based LLM Evaluation (Dev Set): Prompt-based predictions couple LQ and EXP at corr=0.90 versus human corr=0.56, indicating collapse toward one latent quality signal.Few-shot prompting modestly improves LQ to Pearson=0.13 but not EXP, which remains at Pearson=0.01.
  • 6.1. Prompt-Based LLM Evaluation (Dev Set): Development-set prompting and scalar supervision produce near-zero correlations, with LQ r≤0.130 and EXP r≤0.041.The scalar mean objective yields Pearson(s, LQ)=0.092, Pearson(s, EXP)=−0.020, and Pearson(s, combined)=0.050.
  • 6.3. Main results: The dual-head model reaches r=0.388 for LQ and r=0.301 for EXP on the held-out test set, significantly improving over frozen COMET-KIWI.The frozen baseline obtains LQ r=0.219 and EXP r=0.175; bootstrap tests report p<0.05 for the gains.
  • 6.4. Qualitative Error Analysis: Qualitative failures mainly involve multi-step procedural segments where global workflow meaning is preserved but individual steps are omitted or distorted.The model therefore assigns high LQ despite incomplete step-level content coverage.
  • 6.3. Main results: The proposed model produces corr(ŝLQ, ŝEXP)=0.529, close to the human value of 0.56 and unlike prompt-based near-complete coupling.This pattern is presented as evidence that parameter-level disentanglement preserves human-like partial dependence.

7. Discussion

The discussion identifies supervision structure as the main bottleneck and frames the model as a stable-ranking tool for formative assessment under subjective annotation.

  • 7. Discussion: Prompt-based and scalar objectives collapse rubric dimensions, whereas multi-head supervision preserves human-like coupling.The paper targets stable ranking signals rather than absolute agreement, while noting text-only evaluation, moderate dataset size, rater scale variability, and Chinese–English focus.

8. Conclusion

The paper presents a rubric-aligned segment-level SI framework whose LoRA-adapted dual-head COMET-KIWI model improves rubric correlation while preserving human-like coupling.

  • 8. Conclusion: Using 1,101 annotated segments, the framework shows that prompt-based and scalar supervision yield weak alignment with human ratings.The dual-head model improves Pearson correlation to 0.388 for LQ and 0.301 for EXP.
  • 8. Conclusion: The model preserves human-like cross-dimensional coupling, indicating recovery of stable ranking signals under subjective annotation.The stated application is scalable formative feedback and rubric-aligned benchmarking for SI training.
Loading 2609.11131v1…