PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparative Evaluation of Text and Audio Simplification: A Methodological Replication Study

Prosanta Barai, Gondy Leroy, Arif Ahmed

arXiv:2508.15088v1cs.SDeess.AS

TL;DR

This study asks whether text simplification benefits healthcare comprehension when text is presented as audio rather than only in writing. It compares comprehension of audio generated from original and simplified texts, finding that simplification improves performance across measures. The findings support text simplification as a way to improve healthcare communication, while automated free-recall scoring may limit qualitative depth.

  • Problem

    The study examines whether text-simplification benefits for healthcare comprehension extend from written text to audio.

    Source: §5 · p. 9

  • Method

    The study replicated the original comparison by presenting audio generated from original and simplified healthcare texts and measuring perceived difficulty and comprehension.

    Source: §2.1 · p. 3, §2.3 · p. 4

  • Results

    Simplified content improved performance across various metrics, and the benefits of text simplification extended to auditory information.

    Source: §5 · pp. 9–10

  • Takeaways & Limitations

    Text simplification is a practical tool for healthcare communication, while communication strategies may also need to account for education and language proficiency.

    Source: §7 · pp. 11–12

    Automated free-recall scoring may have limited qualitative depth and missed subtle nuances in participants’ responses.

    Source: §6 · p. 11

Abstract

from arXiv · show

This study serves as a methodological replication of Leroy et al. (2022) research, which investigated the impact of text simplification on healthcare information comprehension in the evolving multimedia landscape. Building upon the original studys insights, our replication study evaluates audio content, recognizing its increasing importance in disseminating healthcare information in the digital age. Specifically, we explored the influence of text simplification on perceived and actual difficulty when users engage with audio content automatically generated from that text. Our replication involved 44 participants for whom we assessed their comprehension of healthcare information presented as audio created using Leroy et al. (2022) original and simplified texts. The findings from our study highlight the effectiveness of text simplification in enhancing perceived understandability and actual comprehension, aligning with the original studys results. Additionally, we examined the role of education level and language proficiency, shedding light on their potential impact on healthcare information access and understanding. This research underscores the practical value of text simplification tools in promoting health literacy. It suggests the need for tailored communication strategies to reach diverse audiences effectively in the healthcare domain.

1 Introduction

The study builds on evidence that simplifying healthcare text improves comprehension, and asks whether those benefits carry over to audio. It frames this question against growing use of audio and the importance of clear healthcare communication.

1 Introduction

  • Clear healthcare communication supports health literacy, while poor health literacy can contribute to miscommunication, poor decisions, and higher patient costs.The passage frames health literacy as a national goal in the United States.Source: §1 · p. 2

1.1 Original Study

  • Leroy et al. (2022) found simplified text was perceived as less difficult and improved multiple-choice performance and automated free-recall measures.In their further analysis, education and language spoken at home correlated with accuracy on original texts, with those correlations diminishing for simplified texts.Source: §1.1 · p. 2

1.2 Replication Study

  • This replication tests whether comprehension benefits from simplified healthcare text persist when the information is delivered as automatically generated audio.It extends the original study, which focused on written text.Source: §1.2 · p. 2, §1.2.2 · p. 3

1.2.1 Text to Audio Scripts

  • For analyzable tasks with explicit facts, text and multimedia representations are equally effective at increasing perceived understanding; the written script is also a basis for comprehensible audio and video (Lim & Benbasat, 2000; Schade, 2014).Source: §1.2.1 · p. 2

1.2.2 The Need for Audio Content Simplification

  • Despite growing use of audio technologies for medical information, healthcare audio simplification had not been addressed; the study examines whether it may improve comprehension and compliance.Smart speakers and audio technologies are increasingly used, including by older adults.Source: §1.2.2 · p. 2

2 Materials and Method

The study reused texts simplified in Leroy et al. (2022), generated audio versions, and tested original versus simplified content in a randomized crossover design. It assessed perceived difficulty and comprehension using participant ratings and question-based and automated recall measures.

2.1 Text Simplification

  • Four original and simplified texts covered asthma, liver cirrhosis, pemphigus, and polycythemia vera; the original study’s questions were not simplified.The texts comprised two short and two long texts selected to represent common and less familiar medical topics.Source: §2.1 · p. 3

2.2 Audio Generation

  • Audio was generated with Microsoft Azure text-to-speech, using the same settings across difficulty conditions, including an American English male voice and a 0.95 speech rate.The rate was reported as 145 words per minute, with 1000 ms pauses between sentences.Source: §2.2 · pp. 3–4

2.3 Study Design

  • The randomized crossover design varied original versus simplified text difficulty and measured perceived audio difficulty and actual difficulty through question performance.Participants heard four audios across six order combinations, without repeating a topic; Figure 2 depicts the presentation sequence and combinations.Source: §2.3 · p. 4, Figure 2 · p. 5

2.4 Analytical Measures

  • Comprehension measures included true/false and multiple-choice questions, free recall scored by Word2Vec-based cosine similarity, and ROUGE recall, precision, and F-measures against the audio source text.Perceived difficulty was recorded on a four-point Likert scale; ROUGE assessed longest phrases, unigrams, and bigrams.Source: §2.4 · p. 5

3 Results

Across the replication, simplified audio was perceived as more understandable, while comprehension results varied by question and evaluation measure. The study also compared itemwise actual-difficulty outcomes with the original study.

3.1 Participants and Descriptive Statistics

  • After cleaning, 44 participants and 157 condition-based responses remained from 60 participants; Table 2 reports the retained participants’ demographics.Submissions were excluded for insufficient listening and answering time, incorrect completion codes, or failed attention checks.Source: §3.1 · p. 6, Table 2 · p. 6

3.2 Perceived Difficulty

  • Simplified audio was rated as more understandable than original audio in the replication (2.98 vs 2.52), as in the original study (2.55 vs 2.28).The replication difference was significant (F-value: 14.460, p: 0.001); the original study’s difference was also significant (F-value: 6.130, p: 0.014).Source: §3.2 · p. 7, Table 3 · p. 7

3.3 Actual Difficulty

  • Table 4 presents itemwise one-way ANOVAs comparing actual-difficulty results across the original and replication studies, including means, standard deviations, and significance.The analysis followed the original study’s approach of conducting a one-way ANOVA for each item separately.Source: §3.3 · p. 7, Table 4 · p. 7

3.4 Results Before Listening to Audio

  • Before listening, simplified audio improved accuracy significantly for TF1 (61.64 vs 37.12; p < 0.001), but TF2–TF4 showed no significant differences.The original study found no significant TF1 difference but did find a TF5 difference, which was absent in this replication.Source: §3.4 · p. 8

3.5 Results During and After Listening to Audio

  • During and after listening, simplified audio improved MCQ4 accuracy (71.60 vs 32.42; p = 0.015) and post-listening TF1 and TF3 accuracy.The replication’s significant items differed from the original study, which reported significant improvements for MCQ1, MCQ3, and TF5.Source: §3.5 · pp. 8–9

3.6 Results of Semantic and ML Evaluation

  • For free recall, simplified audio yielded higher similar-word proportions and significant unigram ROUGE recall and F-measure results; cosine similarity was not significant in the replication.The original study reported significant cosine-similarity and ROUGE precision/recall findings, unlike the replication’s cosine result.Source: §3.6 · p. 9

4 Secondary Analysis

The secondary analysis found differing relationships between participant characteristics and answer accuracy, while simplified content improved accuracy across education and language levels.

4 Secondary Analysis

  • Education level was significantly negatively correlated with accuracy (ρ = -0.173, p = 0.030), unlike the original study’s positive correlation.In the replication, accuracy initially increased with education and then consistently decreased; the original study did not observe this trend.Source: §4 · p. 9
  • Accuracy increased with higher English-language proficiency in both studies, while simplified content improved accuracy regardless of education, language, or text/audio length.The education-accuracy pattern described for the replication was not observed in the original study.Source: §4 · p. 9
  • Age, gender, race, ethnicity, and media-consumption groups showed no significant performance differences.The authors report these additional demographic comparisons in Appendix A, Table 1.Source: §4 · p. 9

5 Discussion

The replication found that text simplification improved comprehension in audio, broadly aligning with the written-text study, while question-level differences and listener characteristics qualify the pattern. The discussion highlights modality-specific challenges and the need to consider audience backgrounds when applying simplification.

5 Discussion

  • Simplified content improved performance across various metrics, aligning with the original study and extending its benefits to audio.Figure 3 presents mean accuracy by education, language level, and text for both studies.Source: §5 · pp. 9–10, Figure 3 · p. 10
  • Question-level results largely agree in direction across studies, despite differences in which True/False and multiple-choice items reached significance.The studies found different significant MCQ items; the authors suggest modality-specific processing and recall challenges may contribute.Source: §5 · pp. 10–11
  • Audio's pacing, emphasis, and pauses may shape comprehension differently from self-paced text, complicating modality comparisons.Readers can potentially revisit complex written information, whereas audio presentation may pose information-review challenges.Source: §5 · p. 11
  • Both studies found a nonsignificant performance decrease on the general question with simplified content, possibly reflecting reduced context for broad questions.The authors suggest simplified language may aid specific details while reducing context needed for more general questions.Source: §5 · p. 11
  • Higher education and English proficiency correlated with greater accuracy, while simplification benefited participants across education levels.The authors emphasize tailoring health-literacy interventions to audiences' diverse backgrounds.Source: §5 · p. 11
  • The authors recommend considering delivery modality because simplification's specific benefits may vary by content and assessment method.They call for research on why question types or content areas benefit differently and on mitigating audio's cognitive-load and review challenges.Source: §5 · p. 11

6 Limitations

The study notes limitations from its Azure-generated audio, the absence of advanced-AI benchmarks, and using automated rather than manual free-recall evaluation.

6 Limitations

  • The American English male Azure voice and variable listener preferences constrain how well the audio setup represents diverse comprehension needs.The authors state that these voice and parameter choices did not substantially affect the main comparison of original and simplified audio.Source: §6 · p. 11
  • The study did not benchmark advanced AI simplification models such as ChatGPT because they were not yet widely adopted when the research was conducted.The authors identify evaluation of newer AI advances as future work.Source: §6 · p. 11
  • Without manual evaluation of free-recall responses, automated scoring may have missed subtle qualitative nuances in participants’ answers.The authors chose scalable automated techniques instead of the manual evaluation used in the original study.Source: §6 · p. 11

7 Conclusion

The conclusion reports that simplified healthcare information was perceived as more understandable and improved comprehension, including in audio, while language proficiency and education remain relevant audience factors.

7 Conclusion

  • Simplified healthcare information was perceived as more understandable and improved comprehension, reaffirming the original study’s observations across text and audio.The authors describe text simplification tools as practically useful for promoting health literacy and reaching a broader audience.Source: §7 · p. 11
  • Participants with stronger English proficiency consistently outperformed others, and the analysis highlights education and language proficiency as audience factors for tailored communication.The conclusion calls for communication strategies attuned to target audiences’ needs and backgrounds.Source: §7 · p. 12

Appendix A: Additional Secondary Analysis

The appendix includes a demographic-specific analysis of actual and perceived difficulty.

Appendix A: Additional Secondary Analysis

  • Table 1 is described as a demographic-specific analysis of actual and perceived difficulty, but the supplied passages do not report its results.The available material identifies the table’s scope and the two difficulty categories only.Source: Table 1 · p. 15, Appendix A · p. 15

Appendix B: Text and Questions

Appendix B presents healthcare texts on asthma, cirrhosis, pemphigus, and polycythemia vera, including original and simplified versions, alongside questions assessing perceived difficulty and comprehension. It also includes pre-reading true/false items about polycythemia vera.

Materials

  • The four medical texts and the instrument for assessing perceived and actual difficulty were taken from the original study's appendix.The appendix identifies Leroy et al. (2022) as the source of these materials.Source: Appendix B · p. 15

Asthma text and questions

  • Asthma questions include pre-reading true/false items, perceived-difficulty and overview questions during reading, and post-reading multiple-choice and repeated true/false items.The with-text questions address perceived difficulty, the text's focus on asthma risk factors, and its general account of asthma's causes.Source: Multiple-choice Questions Presented… · p. 17
  • The asthma texts cover atopic and allergen-related risk factors and obesity, with the simplified version restating these topics in more accessible language.Both versions identify atopic history as a strong risk factor, give elevated risks for hay fever and asthma, and discuss the association between obesity and asthma.Source: Original Text · pp. 16–17

Cirrhosis text and questions

  • The cirrhosis texts describe chronic liver damage, common causes, complications, and treatment, while a pre-reading true/false set introduces the topic.The simplified text explains scarring and regenerative nodules, names alcohol, hepatitis, and fatty liver among causes, and describes complications including ascites.Source: Text: Liver cirrhosis · pp. 18–19

Cirrhosis text

  • Cirrhosis content explains how scarring and stellate-cell changes impede liver blood flow, identifies portal hypertension as a major complication, and describes biopsy-based diagnosis.The text also notes that biopsy may be unnecessary when clinical, laboratory, and radiologic evidence indicates cirrhosis, and carries risks.Source: Text: Liver cirrhosis · p. 20

Cirrhosis questions

  • Cirrhosis questions assess perceived difficulty, the text’s overview, complications, links to hepatitis, impaired blood flow, scar formation, and diagnosis.The supplied keyed answers identify liver effects as the overview, fluid retention as a complication, and scar tissue together with liver-cell changes as reducing blood flow.Source: Text: Liver cirrhosis · p. 20

Pemphigus text

  • Pemphigus is described as a group of autoimmune blistering disorders; pemphigus vulgaris can affect both skin and mouth, and untreated disease is usually fatal.The text distinguishes pemphigus vulgaris and foliaceus while noting that some disorders are variably classified as pemphigus and pemphigoid is a separate group.Source: Text: Pemphigus · pp. 21–22

Pemphigus questions

  • Pemphigus questions test the overview, main types, blister locations, autoimmune action, disease classification, and related true/false claims.The keyed choices identify pemphigus vulgaris and foliaceus as the two main types, and blisters on both the outside and inside of the body for pemphigus vulgaris.Source: Text: Pemphigus · p. 22

Pre-reading questions

  • Pre-reading true/false items ask about polycythemia vera’s effects on white blood cells, genetic mutation, and blood viscosity.The supplied items mark the white-cell and viscosity statements false and the genetic-mutation statement true.Source: Text: Polycythemia Vera · p. 23

Appendix B: Text and Questions

  • The paired texts describe polycythemia vera as overproduction of blood cells, especially red cells, which thickens blood and can cause varied symptoms and clots; they also note a common JAK2 mutation.The original and simplified passages both state that the mutation’s role in the disorder is not yet known.Source: Text: Polycythemia Vera · pp. 23–24
  • With the text, participants rated perceived difficulty and answered overview and general questions about bone-marrow blood-cell disorders and polycythemia vera’s classification.The supplied answer key identifies polycythemia vera as one of four myeloproliferative disorders.Source: Text: Polycythemia Vera · p. 24
  • After reading, multiple-choice items tested whether participants understood that excess red cells thicken blood, red-cell increases are greatest, and clots may block blood flow to vital organs.The keyed choices identify thickened blood, more red than white cells, and clot risk as correct.Source: Text: Polycythemia Vera · pp. 24–25
  • A further multiple-choice item tested symptom understanding, with the keyed response noting that symptoms can include features also found in other diseases, such as stomach pains and tiredness.The text also lists nonspecific symptoms such as headaches and fatigue.Source: Text: Polycythemia Vera · pp. 23, 25
  • Repeated true/false items after reading checked claims about white-cell increases, genetic mutation, hyperviscosity, excess red cells, and whether the disorder has only one symptom.The supplied keys mark no white-cell increase and a single symptom as false, while marking the mutation and excess red-cell statements true; hyperviscosity does not mean thin blood.Source: Text: Polycythemia Vera · p. 25

About the Authors

The authors’ backgrounds span healthcare information systems, AI and machine learning, and healthcare analytics. Their research includes using AI to improve healthcare literacy and patient comprehension.

About the Authors

  • Prosanta Barai studies AI and machine learning approaches to healthcare literacy and healthcare delivery, including NIH-funded NLP projects on autism symptom identification.His work emphasizes transparent, interpretable, and ethical AI systems.Source: About the Authors · p. 26
  • Gondy Leroy’s research focuses on machine learning and artificial intelligence, and she directs research at the Center for Management Innovations in Healthcare.She is a Professor of MIS and Associate Dean for Research at the University of Arizona’s Eller College of Management.Source: About the Authors · p. 26
  • Arif Ahmed focuses on healthcare analytics and natural language processing, including AI-based simplification of medical texts to improve patient comprehension.He is working on an NIH-funded project and has experience in machine learning, statistical analysis, and programming.Source: About the Authors · p. 26
Loading 2508.15088v1…