Source-linked AI summary
Intonation Perception in Real and Synthetic Speech across Varying Familiarity Levels: A Pilot Study of Equivalence Assessment
Hanrui Zhou, Gaoyuan Zhang, Yixiang Chen, Yujie Xing, Feng Xu, Xurong Xie, Hui Chen
TL;DR
Language-training corpora are scarce and costly to build, and existing synthetic-speech evaluations have largely focused on declarative sentences despite the relevance of intonation and familiarity. This study compares natural and synthetic speech in similarity-perception and intonation-recognition tasks. Accuracy showed task-specific interactions between speech type and intonation, and between speech type and familiarity.
Problem
High-quality language-training corpora are scarce and costly to construct, while synthetic-speech evaluations have mostly used declarative sentences despite the relevance of intonation and familiarity.
Source: §1 · p. 1Method
The study compares natural and SVC-generated synthetic speech across intonation and familiarity in speaker-similarity and intonation-recognition experiments.
Source: §1 · p. 1Results
Takeaways & Limitations
The findings provide preliminary behavioral evidence for synthetic voices’ perceptual viability in controlled experiments, while corpus construction remains challenging when complex prosody and social cues are involved.
Source: §5 · p. 4, §4.2 · p. 4This is a preliminary exploration, and AI-cloning technology continues to evolve; the study identifies challenges particularly for complex prosody and social cues.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Language training relies on a corpus constructed by a large number linguistic materials. AI-powered voice clones provide a way to construct the corpus with relatively low cost. Singing voice conversion (SVC) model is used to generate synthetic voices. This study compares participants' performances on natural and synthetic speech in two experiments, similarity perception and intonation recognition. In the accuracy of similarity perception task, a significant interaction between speech type and intonation is found, suggesting that question may serve as a cue for speaker identification but may be influenced by synthetic features. In the accuracy of intonation recognition task, a significant interaction between speech type and familiarity is observed, indicating that speech type affects how much familiarity contributes to voice processing.
1. Introduction
The study addresses the scarcity and cost of high-quality language-training corpora by testing whether synthetic voices can support speech processing when intonation and speaker familiarity matter. It compares natural and synthetic speech across two tasks and examines how speech type, intonation, and familiarity relate to performance.
1. Introduction
- High-quality corpora for language training are scarce and costly to build, yet should include speaker identity and prosodic features.Such corpora support the complex cognitive processes involved in language learning.Source: §1 · p. 1
- Most existing synthetic-speech evaluations use declarative sentences, although Mandarin questions may rely on intonation and intonation remains difficult for voice-cloning systems to replicate.Question intonation may be unmarked lexically and can be signaled by a finally rising or high-level f₀ contour.Source: §1 · p. 1
- Familiarity matters because repeated exposure is inherent to training, and familiar voices may facilitate linguistic processing and speaker identification.The passage describes adaptation as a possible account of familiarity effects.Source: §1 · p. 1
- The study compares natural and synthetic voices across statement/question intonation and familiar/strange speakers in similarity-perception and intonation-recognition tasks.It asks how these factors affect performance in each task.Source: §1 · p. 1
- The study’s stated contributions are preliminary support for incorporating AI-generated speech in corpora and exploration of familiar/unfamiliar and natural/AI-generated voice processing.These are framed as practical and theoretical contributions, respectively.Source: §1 · p. 1
2.1. Participant
Seventeen native Mandarin speakers without hearing disorders took part in the study; eleven completed the intonation-recognition experiment.
2.1. Participant
- 17 participants were recruited, and 11 of them took part in intonation recognition.All 17 participated in speaker identity similarity perception.Source: §2.1 · p. 1
- Participants were native Mandarin speakers without hearing disorders and ranged from 22 to 41 years old.The group included 6 women and 11 men.Source: §2.1 · pp. 1–2
- Participants who failed to recognize a familiar speaker were excluded from the speaker identity similarity analysis.They first listened to natural voices from two familiar fellows.Source: §2.1 · p. 2
2.2. Stimuli
The materials paired recorded Mandarin statements and questions from familiar and unfamiliar speakers with synthetic voices generated using an F₀-conditioned SVC model. The synthetic stimuli were evaluated for speaker similarity and selected for intonation expression.
2.2.1. Material
- Eleven neutral Mandarin sentences were recorded as statements and questions by two familiar and two unfamiliar speakers.One sentence was used for practice and the other 10 for the formal experiments; recordings with better quality were selected.Source: §2.2.1 · p. 2
- Figure 1 presents representative f₀ contours for the recorded sentence as a natural statement and a natural question.The speakers were native Chinese women aged 25–45, recorded in a soundproof booth.Source: Figure 1 · p. 2, §2.2.1 · p. 2
2.2.2. AI-generated stimuli
- The F₀-conditioned SVC model captures semantic content and pitch variation from a source singing voice and converts them into a target speaker’s timbre.Synthetic voices were generated with this model.Source: §2.2.2 · p. 2
- The converted voices had mean speaker-similarity scores of 0.55 (SD = 0.28) for statements and 0.51 (SD = 0.30) for questions.Similarity was measured as cosine similarity between speaker embeddings; higher scores indicate greater similarity.Source: §2.2.2 · p. 2
- Higher-scoring synthetic voices were selected, then phonetics experts chose those with the best intonation expression.A third expert made the selection if the first two disagreed; Figure 2 shows representative AI-generated statement and question contours.Source: §2.2.2 · p. 2, Figure 2 · p. 2
2.3. Procedure
The study used a counterbalanced within-subject design across speech type, familiarity, and intonation, with separate similarity-perception and intonation-recognition experiments. Participants completed distinct judgments in each task while listening in a quiet laboratory setting.
2.3. Procedure
- A 2 × 2 × 2 within-subject design crossed speech type, speaker familiarity, and intonation, measuring response time and accuracy.The two experiments were conducted on different days, with counterbalanced stimulus lists and a practice session before each formal experiment.Source: §2.3 · p. 2
2.3.1. Speaker identity similarity perception
- In the similarity task, participants judged whether two sentences came from the same speaker and rated their similarity from 1 to 7.Pairs included same-identity natural voices, natural/synthetic voices of the same identity, and different-identity natural voices, for both familiar and unfamiliar speakers.Source: §2.3.1 · p. 2
- Participants heard voice pairs in balanced, randomized trials and responded after a green fixation cue.The trial procedure is illustrated in Figure 3.Source: §2.3.1 · p. 2, Figure 3 · p. 2
2.3.2. Intonation recognition
- In intonation recognition, participants classified each voice as a statement or question across eight familiarity, speech-type, and intonation conditions.Stimuli were divided into four blocks; participants responded by keyboard after the green fixation cue.Source: §2.3.2 · p. 3, Figure 4 · p. 3
2.3.3. Experimental setting
- Both tasks used headphone listening in a quiet laboratory, with participants seated 65 cm from the computer and able to adjust volume.The monitor had a 60 Hz refresh rate and 1024 × 768 resolution; Figure 5 depicts the setting.Source: §2.3.3 · p. 3, Figure 5 · p. 3
2.4. Data analysis
The two experiments use different statistical models for accuracy, reaction time, and similarity scores, accounting for the data characteristics described for each task.
2.4. Data analysis
- Speaker-identity reaction times were log-transformed and analyzed with linear mixed-effects models; accuracy used binomial generalized mixed-effects models.Similarity ratings used cumulative link mixed models with a logit link because the response scale was ordinal.Source: §2.4 · p. 3
- Models included speech type, familiarity, and intonation as fixed effects, with participants and items represented as random effects.For speaker-identity similarity perception, the score model was a cumulative link mixed model.Source: §2.4 · p. 3
- Intonation-recognition reaction times were log-transformed and analyzed with linear mixed-effects models, while accuracy used Firth’s bias-reduced logistic regression.Accuracy exceeded 95% on average, and reaction time was treated as a more sensitive index of processing differences.Source: §2.4 · p. 3
3. Results
The results show condition-dependent differences across the two experiments: similarity-perception accuracy varied with speech type and intonation, while intonation-recognition accuracy varied with speech type and familiarity.
3.1.1. Accuracy
- Natural–natural pairs were more accurate than natural–synthetic pairs (Mean = 0.917 vs Mean = 0.851), with a question advantage only for natural pairs.The speech-type-by-intonation interaction was significant (Chisq(1) = 5.988, p = 0.014); Figure 6 presents speaker-identity similarity-perception accuracy.Source: Figure 6 · p. 3, §3.1.1 · p. 3
3.1.2. Reaction time
- Strange speakers elicited longer similarity-perception reaction times than familiar speakers (0.848s vs 0.748s); Figure 7 presents reaction time.The familiarity effect was significant (F(1, 1052) = 6.713, p = 0.010).Source: Figure 7 · p. 3, §3.1.2 · p. 3
3.1.3. Score of similarity
- Similarity ratings were higher for strange than familiar speakers (3.930 vs 3.560); Figure 8 presents the similarity scores.Familiarity had a significant effect, while the higher rating for natural–natural speech types versus mixed natural–synthetic types was a main effect reported at p = 0.048.Source: Figure 8 · p. 3, §3.1.3 · p. 3
3.2.1. Accuracy
- Intonation-recognition accuracy showed a significant familiarity-by-speech-type interaction (LR Chisq(1) = 4.078, p = 0.043); Figure 9 presents accuracy.The familiar-versus-strange simple effects were not significant for either natural or synthetic speech.Source: Figure 9 · p. 4, §3.2.1 · p. 4
3.2.2 Reaction time
- Questions elicited marginally longer intonation-recognition reaction times than statements (0.270s vs 0.249s); Figure 10 presents reaction time.The intonation effect was marginally significant (F(1, 1178.03) = 3.844, p = 0.050).Source: Figure 10 · p. 4, §3.2.2 · p. 4
4. Discussion
The discussion reports interactions of speech type with intonation in similarity judgments and with familiarity in intonation recognition. Familiarity also relates to faster responses and lower similarity ratings, while the findings leave current synthetic speech challenging for corpus construction when complex prosody and social cues are involved.
4. Discussion
- Speech type interacted with intonation in similarity-task accuracy and with familiarity in intonation-recognition accuracy.The discussion interprets question intonation as a possible speaker-identification cue that may be influenced by synthetic features.Source: §4 · p. 4, §4.1 · p. 4
4.1. Speaker identity similarity perception
- Question intonation may cue speaker identification, but synthetic voices may further influence this relationship.The discussion relates the cue to prosodic contributions to speaker identification and notes challenges in replicating voice identification.Source: §4.1 · p. 4
- Familiar voices elicited faster responses, while participants gave them lower similarity ratings than strange voices.The authors suggest that familiar acoustic representations support more precise discrimination, so faster responses and lower similarity ratings reflect different aspects of processing.Source: §4.1 · p. 4
4.2. Intonation recognition
- In intonation recognition, the accuracy pattern reversed by speech type: strange voices outperformed familiar voices for natural speech, while the reverse held for synthetic speech.The authors suggest listeners may compare natural familiar voices against stored representations and use different standards for natural and synthetic voices.Source: §4.2 · p. 4
- The reaction-time effect of intonation was marginal, and no interaction was observed, so the effect applied to both real and synthetic speech.The discussion proposes that statement-question f₀ differences may be more discernible at sentence ends.Source: §4.2 · p. 4
- The authors characterize current AI voice cloning as challenging for high-quality training corpora, especially when synthesis includes complex prosody or social cues.They describe this as a preliminary exploration and call for broader, multidimensional investigations as cloning technology evolves.Source: §4.2 · p. 4
5. Conclusions
The conclusion reports interactions involving speech type in both tasks and presents preliminary evidence for SVC-generated voices in controlled experiments. It sees potential for intonation-research corpora while noting that speaker-level prosodic features need further refinement.
5. Conclusions
- Accuracy showed a speech-type interaction with intonation in similarity perception and with familiarity in intonation recognition.The authors suggest prosody may contribute to speaker identification and that participants may treat familiar voices more prudently.Source: §5 · p. 4
- The findings provide preliminary behavioral evidence for the perceptual viability of SVC-generated voices in controlled experimental contexts.The authors suggest SVC voices hold potential for corpus construction in intonation research, with further refinement needed to capture speaker-level prosodic features.Source: §5 · p. 4
- The authors propose larger future studies with more participants, diverse participant groups, and broader corpora.Source: §5 · p. 4
7. Generative AI Disclosure
The disclosure states that generative AI was used only for grammar checking and manuscript polishing.
7. Generative AI Disclosure
- Generative AI was used only for grammar checking and manuscript polishing; the authors reviewed and edited the content and remain accountable for the work.Source: §7 · p. 5