Source-linked AI summary
An Evaluation of AI-Supported Evidence-Based Learning for Public Speaking Skill Development
Sashini Hettiarachchi, Shahbaz Siddeeq, Mika Saari, Pekka Abrahamsson
TL;DR
AI speech coaches often provide generic feedback and lack model examples, despite public speaking’s importance and anxiety burden. This study develops a context-aware coach that generates feedback and revised speech examples in text and audio. In a pilot, results suggest performance improvement and possible anxiety reduction for academic speaking tasks.
Problem
Existing AI speech-coaching applications often overlook speech purpose, audience, and context and lack model speeches, while public speaking is important and anxiety-provoking.
Source: Introduction · p. 1Method
The system evaluates speech using context and the CSSEF, generates revised text and audio examples, and was piloted with repeated practice and pre/post anxiety assessment.
Source: Introduction · p. 2, Experimental Procedure · pp. 4–5Results
Pilot results suggest the system can support improved public speaking performance and may help reduce anxiety in academic speaking tasks.
Source: Conclusion · p. 8, Discussion · p. 8Takeaways & Limitations
Context-aware feedback and model speeches appear to offer practical value for learners needing independent, iterative practice.
Source: Conclusion · p. 8The findings are preliminary because the study had a small sample, short intervention, and no analysis of nonverbal speaking behavior.
Source: Discussion · p. 8
Abstract
from arXiv · showhide
Public speaking is an essential skill in academic and professional contexts, but it often causes anxiety. Although several AI-based speech coaching tools exist, they typically provide generic feedback and lack model speeches for learning. This study addresses these gaps by developing a system that incorporates speech context into evaluation criteria, enabling more tailored feedback. It also generates improved model speeches in both text and audio formats to support example-based learning. The system was evaluated in a pilot study with seven undergraduate students over one to three weeks. Participants used the system repeatedly and practiced the same speech at least three times. Results showed improvements in public speaking performance for all participants. Filler word usage decreased, and anxiety levels dropped by 15.3% to 34.4%. Participants found the context-aware feedback and revised speeches useful and confidence-building. These findings suggest that context-aware feedback and model speeches can enhance public speaking skills and reduce anxiety. Future studies could explore video-based analysis of physical behavior during speeches.
Introduction
Public speaking matters in academic and professional settings but is anxiety-provoking, while traditional practice can be constrained by time, instructor availability, and subjective feedback. This study develops a context-aware AI coach with revised text and audio speech examples, then examines effects on performance, feedback perceptions, and anxiety.
Introduction
- Traditional public speaking classes, organizations, and mentoring offer practice and feedback but can be limited by time, instructor availability, and feedback subjectivity.Public speaking is presented as important in academic and professional development and anxiety-provoking for many people.Source: Introduction · p. 1
- Existing AI speech coaches often give generic feedback that overlooks speech purpose, audience, and context, and commonly lack example speeches for learners.The paper frames both context-aware feedback and model examples as gaps in existing tools.Source: Introduction · pp. 1–2
- The proposed Context-Aware AI Speech Coach evaluates speeches using the CSSEF and generates improved model speeches in text and audio using speech context.Context includes communicative goal and audience.Source: Introduction · p. 2
- The study examines whether context-aware AI feedback and revised speech examples affect public speaking performance, perceived feedback quality, learner perceptions, and anxiety.Its research questions address feedback effectiveness, revised speeches, and anxiety reduction.Source: Introduction · p. 2
Literature review
The literature describes gaps in context awareness and example-based speech learning, alongside established assessment and feedback approaches. Prior work also links preparation and supportive feedback with confidence and reduced public speaking anxiety.
Literature review
- The review describes AI as lacking intent and epistemic awareness, with uneven task performance termed artificial jagged intelligence.The paper focuses on feedback based on user-provided context.Source: Literature review · p. 2
- The CSSEF provides a standardized eight-competency assessment, and this study selected it to compare with Garcia Jr. et al. (2024) while using AI rather than a teacher for evaluation.The form rates competencies as inadequate, satisfactory, or exceptional.Source: Literature review · pp. 2–3
- Many AI speech-training tools assess audio or video through surface-level delivery metrics while overlooking audience, speech goal, type, and subject context.MySpeaker Rhetorich could assess pacing and pitch but did not consider speech subject, context, or target audience.Source: Literature review · p. 3
- Prior work describes model examples as useful for speech learning, while Garcia Jr. et al. (2024) offered limited guidance on constructing and revising speeches.VoiceCoach recommended expert vocal-modulation samples (Wang et al., 2020); repeated audio exposure was also associated with stronger listening, pronunciation, and fluency.Source: Literature review · p. 3
- Prior studies report that AI feedback tailored to student performance can support targeted learning outcomes and can be as effective as human-generated feedback.These findings are attributed to Ba et al. (2025) and Kaliisa et al. (2025).Source: Literature review · p. 3
- Preparation and supportive feedback have been reported as strategies associated with reduced public speaking anxiety and increased confidence.Preparation was reported as a mitigation strategy (Mohd Sobri Paridaluddin et al., 2023), and public speaking courses were reported to reduce fear and build confidence.Source: Literature review · p. 3
Overview of the experiment
The pilot involved seven University of Moratuwa undergraduates who practiced the same speech repeatedly over one to three weeks alongside their regular academic commitments.
Overview of the experiment
- Seven undergraduates at the University of Moratuwa, Sri Lanka, aged 21–23, participated; all spoke English as a second language.The group included three female and four male students.Source: Overview of the experiment · p. 4
- Participants used the AI speech coach multiple times and practiced the same speech at least three times, completing the experiment in one to three weeks.Some finished three practice sessions in one week, while others took up to three weeks.Source: Overview of the experiment · p. 4
- Regular academic commitments limited how much time participants could devote exclusively to speech practice.Each participant was assigned an identifier from P1 to P7 to preserve anonymity.Source: Overview of the experiment · p. 4
Speech Analysis Process
The system analyzes uploaded speech audio and produces CSSEF-based feedback and revised model speeches. Participants completed baseline and post-practice anxiety measures and iteratively reviewed, reflected on, and applied the AI feedback and speech examples.
Speech Analysis Process
- Uploaded audio is transcribed, checked for filler-word frequency, and analyzed for prosodic features before competency evaluation and feedback generation.The pipeline extracts pitch variation, speech rate, pauses, and volume changes, and identifies fillers such as “um,” “uh,” and “like.”Source: Speech Analysis Process · p. 4, Figure 1 · p. 5
- The system evaluates seven CSSEF competencies with GPT-4o on a 1–5 scale, excluding Physical Behaviors (C8) because the implementation supports audio and text analysis only.It reports an overall percentage score, strengths, and areas for improvement.Source: Speech Analysis Process · p. 4
- After each submission, the system generates revised text using user context and CSSEF results, plus audio based on that text as a model of clearer articulation, pacing, vocal variety, and delivery.The text revision focuses on structure, clarity, organization, and flow.Source: Speech Analysis Process · p. 4
Experimental Procedure
- Participants completed a baseline PRPSA after a 30-minute online orientation and later repeated it after practice, followed by a questionnaire about feedback, revised speeches, and perceived anxiety changes.The pre-experiment survey included consent and background questions.Source: Experimental Procedure · pp. 4–5
- Participants selected an academic topic and repeatedly reviewed the AI feedback and revised text and audio, reflected on differences, and revised and re-recorded the same speech.Speeches required a title, purpose, intended audience, and key points, with a target duration of 2–4 minutes.Source: Experimental Procedure · p. 5
Results and analysis
First-to-final CSSEF scores showed substantial gains for some participants, with P5 improving across all seven competencies. Gains were also reported by baseline anxiety group, though no high-anxiety participants were included.
Effectiveness of Context Aware AI Generated Feedback
- P2, P5, and P7 showed substantial, consistent first-to-final CSSEF gains; P5 improved across all seven competencies, with the largest relative gain at +120%.Table 1 presents the overall score changes.Source: Table 1 · p. 5, Effectiveness of Context Aware AI… · p. 5
- Participants with moderate PSA improved CSSEF scores by +28.8%, while the low-PSA participant improved by +16.1%.No participants were classified as high PSA at baseline; Garcia Jr. et al. (2024) reported CSSEF improvements of 54–70% over a structured multi-day intervention.Source: Effectiveness of Context Aware AI… · p. 6
Impact of Revised Speeches
Participants showed improvements in delivery-related competencies and reduced filler-word use. They also rated the revised speeches favorably, with every usefulness question scoring above 4 out of 5.
Impact of Revised Speeches
- All revised-speech usefulness questions scored above 4 out of 5 among the seven participants.Table 3 summarizes participants’ ratings on the revised speeches.Source: Table 3 · p. 6, Impact of Revised Speeches · p. 7
- Filler-word usage fell for all participants, with relative decreases of 25% to 100%; most also gained +1 in Vocal Variety, and six of seven improved in Organization.Organization gains ranged from +1 to +2; improvements in Vocal Variety aligned with use of AI-generated audio examples.Source: Impact of Revised Speeches · pp. 6–7
Effect of AI Speech Coaching Application on Public Speaking Anxiety
All seven participants had lower PRPSA scores after using the AI Speech Coach, with percentage reductions ranging from 15.3% to 34.4%. Reductions were reported for both baseline anxiety groups represented in the study, but none had high anxiety at baseline.
Effect of AI Speech Coaching Application on Public Speaking Anxiety
- PRPSA scores fell for all seven participants by 19 to 45 points, corresponding to reductions of 15.3% to 34.4%.Six participants moved from moderate to low anxiety; the remaining participant stayed in the moderate category but also had a measurable reduction.Source: Table 4 · p. 7, Effect of AI Speech Coaching… · p. 7
- Participants with moderate baseline anxiety (n = 6) had an average PRPSA reduction of 33.5 points (28.8%); the low-anxiety participant’s score fell by 29.1%.No participants were classified as high anxiety at baseline.Source: Table 5 · p. 7, Effect of AI Speech Coaching… · pp. 7–8
- The study’s percentage reductions, particularly for the moderate-anxiety group, were comparable or slightly higher than Garcia Jr. et al. (2024), despite differences in sample size and intervention conditions.Garcia Jr. et al. reported PRPSA reductions of 23.3% to 26.9% across low, moderate, and high anxiety groups.Source: Effect of AI Speech Coaching… · p. 8
Discussion
The pilot findings suggest the context-aware AI Speech Coach may support academic speaking practice and reduce anxiety, but the evidence remains preliminary. The study’s small sample, short duration, and lack of nonverbal analysis constrain interpretation.
Discussion
- The system was useful as a formative tool for academic speaking, with performance gains particularly in content clarity and organization.Revised text and audio speeches were perceived as helpful for reflection and revision.Source: Discussion · p. 8
- Decreased PRPSA scores indicate that structured, self-paced AI-supported practice may help reduce public speaking anxiety.Source: Discussion · p. 8
- The findings are preliminary because the study had a small sample, a short intervention, and no analysis of posture, gesture, or eye contact.The authors call for larger samples, longer interventions, and multimodal analysis in future research.Source: Discussion · p. 8
Conclusion
The study presents a context-aware AI Speech Coach that provides feedback and revised speech examples in text and audio. Pilot findings suggest educational potential for independent, iterative practice, but do not establish definitive effectiveness.
Conclusion
- The pilot suggests the system can support improved public speaking performance and may help reduce anxiety in academic speaking tasks.Source: Conclusion · p. 8
- The coach combines contextualized feedback with revised speech examples in text and audio formats.Source: Conclusion · p. 8
- The authors frame the contribution as demonstrating feasibility and educational potential, not definitive effectiveness, given the small sample and limited study duration.The approach aims to make practice more meaningful, targeted, and accessible by incorporating speech context and concrete examples.Source: Conclusion · p. 8