A barrier or a booster? Familiarity effects on Mandarin emotion prosody recognition using AI-powered voice cloning

1. Introduction

• Emotion prosody perception requires listeners to process acoustic cues alongside social indices such as speaker identity and relationship.The cited passage gives pitch and speech rate as examples of acoustic cues.Source: §1 · p. 1
• AI emotional speech can contain unnatural coarticulation, spectral distortions, and prosodic micro-perturbations despite advances in voice synthesis.These atypicalities are described as potentially disrupting the transmission of emotional intent.Source: §1 · p. 1
• Prior work reports higher accuracy and shorter reaction times for natural human emotional speech than AI-synthesized speech, motivating an isolated acoustic comparison.The introduction notes the need to test whether AI prosody can reach human recognition levels when visual and complex contextual cues are absent.Source: §1 · p. 1
• Familiarity may supply prosodic memory that supports emotion recognition, but its effects on AI-cloned familiar voices remain unexplored.The introduction frames the competing possibilities as emotional-memory transfer or an uncanny-valley response to mismatched cues.Source: §1 · pp. 1–2
• The study asks whether AI emotion recognition and cognitive load differ from human speech, and how speaker familiarity changes perception of AI prosody.The questions focus on speech isolated from visual and contextual cues.Source: §1 · p. 2

2.1. Participant

• Seventeen Mandarin-speaking adults aged 22–41 years (M = 28.18, SD = 26.80) participated; none reported hearing or cognitive impairments, and the sample size was described as consistent with standard within-subject designs in this field.Source: §2.1 · p. 2

2.2.1. Materials

• Fourteen neutral Mandarin sentences were paired with four emotions; emotional recordings from a broadcasting student supplied the stranger-voice stimuli and AI-synthesis source.Neutral recordings from a familiar speaker served only as target timbres because that speaker could not reliably produce the target emotional prosody.Source: §2.2.1 · p. 2

2.2.2. AI voice cloning

• An adapted F0-conditioned singing-voice-conversion model transferred source semantic content and emotion-related pitch contours to a target timbre.The effort factor controls arousal transfer, and experts reviewed and filtered generated samples before use.Source: §2.2.2 · p. 2

2.3. Procedure

• Participants judged each sentence’s emotional category in a pseudo-randomized, three-block task after confirming recognition of familiar speakers.The study also collected continuous physiological recordings during the task.Source: §2.3 · p. 2

2.4. Analysis

• Accuracy, reaction time, and HRV features were analyzed, with mixed-effects models and Tukey-adjusted post-hoc comparisons for significant multilevel effects.HRV features included mean heart rate, RMSSD, LF, HF, and the LF/HF ratio.Source: §2.4 · pp. 2–3

3.1. Human vs. AI-synthesized voice

• Accuracy was significantly higher for human than AI voices across emotions except fear.The source analysis was restricted to stranger voices.Source: §3.1 · p. 3, Table 2 · p. 3, Figure 1 · p. 3
• Reaction times were significantly shorter for human than AI voices across emotions except fear.The source analysis found a significant Source × Emotion interaction.Source: Figure 2 · p. 3, Table 4 · p. 3, Table 3 · p. 3
• Source-based HRV models did not fit significantly better than their corresponding null models (all ps > .05).The analysis tested whether HRV features discriminated between speech sources.Source: Figure 3 · p. 3

3.2. Effects of familiarity

• For happy stimuli, accuracy was significantly higher for familiar than stranger AI voices (β = 0.66, SE = 0.13, z = -2.098, ps = .036); other emotions showed no significant familiarity differences.The familiarity analysis was restricted to AI voices.Source: §3.2 · p. 3, Figure 4 · p. 3, Table 5
• Fearful stimuli elicited significantly faster responses for stranger than familiar voices (β = -0.19, SE = 0.07, z = -2.810, ps = .005); other emotions showed no significant familiarity differences.The familiarity analysis was restricted to AI voices.Source: §3.2 · p. 3, Figure 5 · p. 4, Table 6 · p. 4
• Familiarity-based HRV models did not fit significantly better than their corresponding null models (all ps > .05).The analysis tested whether HRV features discriminated between familiar and stranger voices.Source: Figure 6 · p. 4

• Human voices yielded significantly higher accuracy and shorter reaction times than AI voices for happiness, sadness, and anger, but not fear.HRV did not significantly distinguish speech sources.Source: §4 · p. 4
• The authors interpret slower responses to AI speech as greater processing demand from synthetic cues that deviate from natural human prototypes.Source: §4 · p. 4
• Familiar AI voices improved accuracy for happy stimuli, whereas fearful AI stimuli prompted faster responses from stranger than familiar voices.The authors suggest the latter delay may reflect an uncanny valley effect; familiarity did not significantly affect HRV.Source: §4 · p. 4
• The findings frame synthetic emotion decoding as gated by top-down social cognition and speaker identity, with familiar AI voices producing different responses across emotional valences.The authors identify implications for cognitive models of human-machine interaction and note that developers should account for cognitive load imposed by synthetic speech.Source: §4 · p. 4

• Human voices afforded higher emotion-recognition accuracy and faster processing than AI voices.Source: §5 · p. 4
• Familiarity with an AI voice provided a compensatory advantage for positive emotions but triggered critical evaluation delays for negative emotions.The authors attribute the delays to the uncanny valley effect.Source: §5 · p. 4
• The study integrated behavioral and physiological measures to investigate how speech source and speaker familiarity affect emotion prosody perception.Source: §5 · p. 4

• The authors used Gemini only for grammar and readability checks, reviewed and edited the content, and retained full responsibility for the publication.Source: §7 · p. 5