PaperlayerView exact revision on arXiv

Source-linked AI summary

How Should We Evaluate Naturalness for Synthetic Speech with a Foreign Accent?

Kentaro Onda, Satoru Fukayama, Daisuke Saito, Nobuaki Minematsu

arXiv:2610.13175v1cs.SDeess.AS

TL;DR

The paper examines whether foreign accent affects human naturalness judgments, given that UTMOS tends to score stronger accents lower and human judgments may also depend on instructions and listener backgrounds. In a large-scale subjective evaluation, human-likeness instructions mitigated accent effects, while exposure to the target accent appeared to help raters distinguish accent from system-induced unnaturalness.

  • Problem

    It was unclear whether lower naturalness judgments for stronger foreign accents were specific to automatic MOS prediction or also occurred in human evaluation, whose judgments can depend on instructions and evaluator characteristics.

    Source: §I · pp. 1–2

  • Method

    The study evaluated analysis-resynthesized ERJ speech with human raters under varied instructions and compared the findings with automatic evaluation systems.

    Source: §III · p. 2, §I · p. 2, B. Subjective evaluation design · p. 3

  • Results

    Human listeners also associated accentedness with reduced naturalness; human-likeness instructions mitigated that influence while making system-level differences more observable, and target-accent exposure appeared to help separate accent from system-induced unnaturalness.

    Source: §V · p. 6

  • Takeaways & Limitations

    Appropriate evaluation of accented synthetic speech calls for carefully designed instructions and raters familiar with the target accent.

    Source: §V · p. 6

    As speech targets diversify, conventional naturalness criteria and crowdsourced listening may be insufficient; evaluation may require context-specific protocols and listeners with relevant expertise.

    Source: §V · p. 6

Abstract

from arXiv · show

Recent advances in speech synthesis technology have enabled the generation of speech with diverse characteristics. One such characteristic is a foreign accent. In our preliminary experiments, we found that UTMOS, a widely used automatic speech quality assessment system, tended to assign lower scores to speech with stronger foreign accents, even though it is produced by a human speaker. This observation raised a question: How do humans evaluate the naturalness of foreign-accented speech? We conducted a large-scale subjective evaluation involving both native and non-native listeners under different evaluation instructions. The results showed that humans also considered accentedness when evaluating naturalness, but that defining naturalness in terms of human-likeness and selecting raters who have greater exposure to the target accent may enable more appropriate evaluation. This study aims to provide a starting point for future discussions on fairer and more appropriate evaluation of synthetic speech with a foreign accent.

I. INTRODUCTION

The paper asks whether foreign accent lowers perceived naturalness for human listeners as it does for UTMOS, and whether instructions and rater backgrounds shape that judgment. Its findings indicate that human-likeness instructions and raters familiar with the target accent may support more appropriate evaluation.

I. INTRODUCTION

  • UTMOS tended to assign lower naturalness scores to less-proficient Japanese speakers of English, although their utterances were produced by human speakers.The preliminary experiment related speaker proficiency to average UTMOS scores; the figure reports lower predicted scores for speakers with lower proficiency.Source: §I · p. 1, Figure 1 · p. 1
  • Because foreign accent can be an intended synthesis characteristic, accent-linked score reductions may misrepresent system quality and implicitly favor native-accented speech.The paper notes that MOS predictors are also used for data filtering, corpus selection, and reinforcement-learning rewards.Source: §I · p. 1
  • It remained unclear whether accent-linked naturalness judgments were specific to automatic predictors or also reflected human judgments shaped by instructions and evaluator characteristics.The authors motivate examining both automatic and human evaluation because naturalness is ambiguous and related judgments can depend on listeners.Source: §I · pp. 1–2
  • The study investigates human and automatic evaluation, asking whether stronger foreign accents lower human ratings and how instructions and evaluator backgrounds affect that relationship.Its large-scale subjective evaluation involved 320 participants across multiple evaluation instructions and rater groups.Source: §I · p. 2
  • Human listeners also considered accentedness when judging naturalness; human-likeness instructions reduced its influence while preserving sensitivity to system-level quality differences.Raters with greater exposure to the target accent appeared better able to distinguish accent-related characteristics from system-induced unnaturalness.Source: §I · p. 2

II. DEFINITION OF NATURALNESS

The paper treats naturalness as ambiguous and examines two perspectives: human-likeness and context conformity. Context conformity brings together application-specific appropriateness and deviation from a specified reference.

II. DEFINITION OF NATURALNESS

  • Naturalness has been criticized as ambiguous, while traditional MOS evaluation has remained an effective assessment method despite that ambiguity.The paper notes that researchers have begun reconsidering naturalness by separating it into finer components.Source: §II · p. 2
  • One account distinguishes human-likeness from appropriateness, which evaluates whether speech meets application-specific requirements or social norms.Examples of application contexts include fictional characters, robots, and settings where gender or accent inclusivity is required.Source: §II · p. 2
  • Deviation-based naturalness measures departure from an explicitly specified reference, which may be speech samples or a rater’s internal representation.The paper groups this perspective and appropriateness under context conformity: matching expectations set by a context or reference.Source: §II · p. 2

III. DATA COLLECTION: HOW DO HUMANS EVALUATE SYNTHETIC SPEECH WITH A FOREIGN ACCENT?

The study used analysis-resynthesized speech from native and non-native ERJ corpus speakers in a subjective evaluation. Participants rated samples under varied conditions, yielding 15,360 annotations.

III. DATA COLLECTION: HOW DO HUMANS EVALUATE SYNTHETIC SPEECH WITH A FOREIGN ACCENT?

  • Evaluation samples were generated by analysis-resynthesis of native and non-native speech from the ERJ corpus.The experiment investigated how humans perceive and evaluate the naturalness of foreign-accented synthetic speech.Source: §III · p. 2
  • A total of 320 participants provided 15,360 annotations for 480 speech samples.Human raters evaluated each sample under a variety of conditions.Source: §III · p. 2

A. Speech Stimuli

The stimulus set used ERJ recordings from Japanese learners of English and native General American speakers. Restoration and three analysis-resynthesis systems produced four versions of each original recording.

A. Speech Stimuli

  • Five phoneme-coverage-oriented sentences from ERJ’s S_PH_B_1 set were selected, with recordings from 20 Japanese English learners and four General American speakers.The source set comprised 120 original samples.Source: A. Speech Stimuli · pp. 2–3
  • Restoration was applied to all recordings to reduce recording-condition differences; DNSMOS scores for Japanese English and American English became nearly identical after restoration.Before restoration, Japanese English had lower DNSMOS than American English; the UTMOS–proficiency correlation also became stronger after restoration.Source: A. Speech Stimuli · p. 3, Table I · p. 3
  • Each original recording had four versions: restoration only, BigVGAN, EnCodec, and WORLD, producing 480 stimuli resampled to 16,000 Hz.Source: A. Speech Stimuli · p. 3

B. Subjective evaluation design

The study varied how listeners were asked to interpret naturalness, contrasting human-likeness with context-conformity instructions. It used a five-point rating scale and recruited native English and Japanese listeners.

B. Subjective evaluation design

  • The four instructions asked for general naturalness, human-likeness, or naturalness assuming the speaker was native or non-native.The general instruction was the most ambiguous; the other instructions specified an interpretation of naturalness.Source: B. Subjective evaluation design · p. 3
  • Listeners rated samples on a five-point scale labeled Bad, Poor, Fair, Good, and Excellent.The labels corresponded to scores 1 through 5.Source: B. Subjective evaluation design · p. 3
  • The test included 15,360 ratings from 320 listeners: 160 native English speakers and 160 native Japanese speakers.Participants were recruited for each instruction–subgroup combination, with quality-control samples used to screen responses.Source: B. Subjective evaluation design · p. 3

IV. DATA ANALYSIS

Human ratings favored American English over Japanese-accented English across systems, instructions, and rater groups. Human-likeness instructions narrowed that gap while preserving clearer system-level differences than the context-based instructions.

A. Overall results

  • Across systems, instructions, and rater groups, AE received higher ratings than JE, including when listeners judged naturalness as non-native speech.JE also had higher standard deviations than AE in most cases.Source: A. Overall results · p. 4, Table III · p. 4
  • Human-likeness instructions generally raised JE ratings and lowered AE ratings relative to general naturalness, reducing the AE–JE rating gap.Instructions to judge naturalness as native speech lowered JE ratings, while non-native-speech instructions lowered AE ratings, relative to general naturalness.Source: A. Overall results · p. 4, Table III · p. 4
  • Japanese native raters consistently rated JE higher than English native raters across instructions and systems, while AE ratings were generally similar under general-naturalness and human-likeness instructions.The authors suggest Japanese raters’ greater exposure to Japanese-accented English may contribute to the higher JE ratings.Source: A. Overall results · p. 4, Table III · p. 4
  • System rankings were largely consistent across rater groups and instructions: No resyn. and BigVGAN generally rated highest, followed by Encodec and WORLD.Some conditions produced smaller rating gaps among systems, indicating that system-quality discriminability depended on task framing.Source: A. Overall results · p. 4, Table III · p. 4

B. Relationship between accent strength and perceived naturalness

Within JE speech, higher proficiency was associated with higher naturalness ratings across conditions, but the strength of the association varied with instructions and rater background. Human-likeness and non-native-speech framing reduced the association without eliminating it.

B. Relationship between accent strength and perceived naturalness

  • Positive proficiency–MOS correlations appeared in every condition, meaning speakers with weaker accents tended to receive higher naturalness ratings.The correlation strength varied by instruction and listener background.Source: B. Relationship between accent… · p. 4, Table IV · p. 5
  • Correlations were generally weaker under human-likeness instructions than under general-naturalness instructions, consistent with reduced influence of accentedness on ratings.Under human-likeness, native English raters showed higher correlations than native Japanese raters.Source: B. Relationship between accent… · p. 4, Table IV · p. 5
  • Non-native-speech framing generally produced the weakest proficiency–MOS correlations, but positive correlations remained in every condition.The framework therefore reduced, but did not eliminate, the effect of speaker proficiency on perceived naturalness.Source: B. Relationship between accent… · pp. 4–5, Table IV · p. 5

C. Can humans evaluate systems beyond foreign accent?

Human-likeness was the most suitable of the tested criteria for judging foreign-accented synthetic speech: it reduced accent influence and revealed system-level differences most clearly. Context-conformity instructions made those differences less apparent.

C. Can humans evaluate systems beyond foreign accent?

  • System-level differences among JE speech were clearest under human-likeness instructions, which reduced the influence of foreign accent.Under native- and non-native-context instructions, system differences became less apparent, particularly among Japanese raters.Source: C. Can humans evaluate systems beyond… · p. 5
  • The weaker proficiency–MOS correlations under non-native-speech instructions did not mean listeners focused more on system quality; accent could draw attention while system differences became less apparent.The authors note that stronger perceived accent was not necessarily associated with lower naturalness under this framing.Source: C. Can humans evaluate systems beyond… · p. 5
  • Japanese raters, who had greater exposure to Japanese-accented English, may have been better able to distinguish accent-related traits from system-induced unnaturalness when judging human-likeness.Source: C. Can humans evaluate systems beyond… · p. 5

D. How do existing automatic evaluation systems evaluate foreign-accented synthetic speech?

The three MOS prediction systems generally rated AE speech higher than JE speech, consistent with the human evaluations, but their proficiency trends differed. Their strongest correlations with human ratings were under the human-likeness instruction.

D. How do existing automatic evaluation systems evaluate foreign-accented synthetic speech?

  • All three models assigned higher scores to AE than JE, while their score trends with JE speaker proficiency varied across models.UTMOS correlated positively with proficiency, whereas SSL-MOS showed a negative correlation.Source: D. How do existing automatic… · pp. 5–6, Figure 3 · p. 6
  • All three models had their highest correlations with human ratings under the human-likeness instruction.The correlations are utterance-level Spearman coefficients for JE speech (N=400).Source: D. How do existing automatic… · p. 6, Table V · p. 6
  • System-level comparisons differed by model: all system pairs differed significantly for UTMOS and SSL-MOS, but EnCodec and WORLD did not differ significantly for UTMOSv2.The reported significance for UTMOS and SSL-MOS was p < 0.001.Source: D. How do existing automatic… · p. 6

V. DISCUSSION: IMPLICATIONS FOR EVALUATING ACCENTED SPEECH SYNTHESIS

Human ratings reflected accentedness: JE speech scored lower than AE speech, and proficiency within JE speech was positively associated with MOS. Human-likeness instructions reduced, but did not eliminate, this influence, underscoring the importance of instructions and rater familiarity.

V. DISCUSSION: IMPLICATIONS FOR EVALUATING ACCENTED SPEECH SYNTHESIS

  • Across conditions, JE speech received lower ratings than AE speech, and JE proficiency was positively correlated with MOS.The human-likeness instruction mitigated the influence of accentedness, but the effect was incomplete; native English raters showed a stronger tendency than Japanese raters to lower scores for stronger accents.Source: §V · p. 6
  • Listeners less familiar with the target accent may interpret accent-related phonetic and prosodic traits as system-induced unnaturalness.The authors suggest that appropriate evaluation requires carefully designed instructions and raters familiar with the target accent.Source: §V · p. 6
  • The controlled analysis-resynthesis setting largely held accent degree across systems and retained stimulus characteristics from real speech, unlike potentially more complex accented TTS.For TTS, accent degree may vary by system and synthesis may add segmental or prosodic unnaturalness, making accent-versus-defect attribution harder.Source: §V · p. 6
  • Future evaluations may need target-accent exposure and expertise, as diverse synthesis targets challenge shared naturalness criteria and crowdsourced assessment.The authors emphasize distinguishing accent-related characteristics from system-induced unnaturalness.Source: §V · p. 6

VI. CONCLUSIONS

The study used a large-scale subjective evaluation with native English and Japanese listeners and multiple instructions. Its analysis points to instruction design and rater selection as important for evaluating accented synthetic speech.

VI. CONCLUSIONS

  • The subjective evaluation involved 320 native English and Japanese raters and collected 15,360 five-point ratings.The evaluation examined human-likeness and context conformity using four instructions.Source: §VI · p. 6
  • The analysis highlighted the importance of evaluation instructions and rater selection for evaluating accented synthetic speech appropriately.Source: §VI · p. 6
  • The authors note limitations in evaluations based simply on “naturalness” and those relying solely on crowdsourced listening tests.They present the analyses and discussion as a starting point for further discussion of evaluation.Source: §VI · p. 6
Loading 2610.13175v1…