PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Accessibility of Medical Texts through Large Language Model-Driven Plain Language Adaptation

Ting-Wei Chang, Hen-Hsen Huang, Hsin-Hsi Chen

arXiv:2609.17398v1cs.CL

TL;DR

Complex medical language can exceed patients’ reading comprehension, creating a need for accessible automated PLA. This study compares prompting, QLoRA finetuning, and MoA across LLMs, finding that these approaches demonstrate potential for simplifying medical texts while preserving essential content. The study is limited by reliance on the PLABA dataset and general-purpose models rather than medical-domain pretraining.

  • Problem

    Complex healthcare language can be difficult for the public to understand, motivating PLA to bridge medical texts and patients’ reading comprehension.

  • Method

    The study compares prompting strategies, QLoRA finetuning, MoA, automated evaluation, and LLM judging across several LLMs for PLA.

  • Results

    LLM-driven PLA demonstrates potential for simplifying complex medical texts while producing relevant and readable outputs.

  • Takeaways & Limitations

    Few-shot semantic-similarity prompting, finetuning, and MoA provide supported approaches for adapting general-purpose LLMs to PLA tasks.

  • Takeaways & Limitations

    The study uses only the PLABA dataset, which may not represent the full range of medical texts encountered in real-world applications.

Abstract

from arXiv · show

This paper addresses the challenge of making complex healthcare information more accessible through automated Plain Language Adaptation (PLA). PLA aims to simplify technical medical language, bridging a critical gap between the complexity of healthcare texts and patients' reading comprehension. Recent advances in Large Language Models (LLMs), such as GPT and BART, have opened new possibilities for PLA, especially in zero-shot and few-shot learning contexts where task-specific data is limited. In this work, we leverage the capabilities of LLMs such as GPT-4o-mini, Gemini-1.5-pro, and LLaMA for text simplification. Additionally, we incorporate Mixture-of-Agents (MoA) techniques to enhance adaptability and robustness in PLA tasks. Key contributions include a comparative analysis of prompting strategies, finetuning with QLoRA on different LLMs, and the integration of MoA technique. Our findings demonstrate the effectiveness of LLM-driven PLA, showcasing its potential in making healthcare information more comprehensible while preserving essential content.

1 Introduction

Medical texts often exceed patients’ reading comprehension, motivating PLA to simplify healthcare information while preserving accessibility. This paper compares prompting, QLoRA finetuning, MoA, and automated evaluation approaches for LLM-driven PLA.

  • PLA addresses the gap between complex medical language and patients’ reading comprehension by simplifying healthcare texts.The goal is to make vital health information more accessible.
  • The study compares zero-shot prompting, in-context learning, and semantic-similarity-enhanced in-context learning.
  • QLoRA finetuning is evaluated for its effect on generating simplified medical language with advanced LLMs.
  • Mixture-of-Agents combines capabilities from multiple LLMs to assess its impact on PLA performance.
  • Automated metrics and LLM judging are combined to make output analysis more comprehensive and reliable.
  • The experiments evaluate the new PLABA 2024 term replacement task using varied prompting techniques.

2 Related Work

Related work establishes datasets and model-based approaches for simplifying biomedical texts and producing lay summaries. Prior studies examine specialized finetuning, prompting, efficient adaptation, and collaborative multi-LLM architectures.

  • Manual plain-language adaptation does not scale to every medical article, motivating automated adaptation with language models.
  • PLABA provides expert-annotated, sentence-aligned biomedical abstract pairs for document- and sentence-level simplification.The dataset includes 7,643 aligned pairs and sentence splitting for readability.
  • Earlier PLABA systems used LLaMA 2 and addressed limited substantive editing through sentence- and token-level loss weighting.
  • Prior biomedical simplification work compared T5, SciFive, BART, and GPT models using domain-specific finetuning and prompt-based learning.BART-Large with a control token achieved the highest SARI score, while T5-Base scored best on BERTScore.
  • BioLaySumm differs from PLABA by targeting abstractive, accessible summaries of full biomedical articles for non-expert readers.
  • QLoRA reduces finetuning memory and computation through 4-bit quantization and low-rank adapters without unfreezing pretrained weights.
  • MoA organizes multiple LLMs into layers so later agents receive outputs from earlier agents, leveraging collaborative generation.

3 Methodologies

The methodology evaluates LLM-based PLA through multiple prompting and finetuning settings, alongside a separate term replacement pipeline. The term replacement process identifies difficult terms, classifies replacement types, and generates lay equivalents.

  • PLA Task: The PLA task simplifies abstracts while keeping sentences separate and preserving accuracy and clarity for general readers.
  • PLA Task: PLA experiments compare zero-shot prompting, random few-shot ICL, semantically similar few-shot ICL, and zero-shot inference after QLoRA finetuning.
  • PLA Task: Figure 1 depicts the few-shot PLA pipeline, including annotated-data splitting into training and validation sets.
  • Term Replacement Task: The term replacement task identifies complex terms, classifies the needed replacement type, and generates lay-language equivalents using semantically enhanced few-shot ICL.

3.1 Models

The study uses a diverse set of LLMs with different capabilities and efficiency characteristics. The model set includes Gemini, GPT-4o-mini, LLaMA, Gemma, and Mistral families.

  • The experiments use diverse advanced LLMs selected for capabilities relevant to natural-language processing.
  • Gemini: Gemini models support multimodal inputs, while Gemini-1.5 models are designed for efficient processing of long contexts.
  • GPT-4o-mini: GPT-4o-mini is optimized for natural-language performance while reducing cost and latency.
  • LLaMA: LLaMA model variants provide multilinguality, reasoning, and long-context processing, with LLaMA 3.1 supporting up to 128K tokens.
  • Gemma 2: Gemma 2 is a lightweight family spanning 2B, 9B, and 27B parameters, using attention and distillation techniques to support smaller models.
  • Mistral: Mistral 7B uses grouped-query and sliding-window attention to improve inference speed and support longer sequences.

3.2 QLoRA

QLoRA finetuning is configured to reduce the computational and memory demands of adapting models for PLA.

  • 3.2 QLoRA: 4-bit quantization reduces the computational load during QLoRA finetuning.The setup uses LoRA rank 16 and alpha 16.
  • 3.2 QLoRA: QLoRA finetuning uses a learning rate of 2e-4, batch size 2, and one training epoch.

3.3 MoA

The paper applies a single-layer Mixture-of-Agents framework to PLA, using Gemini-1.5-Flash for aggregation and synthesis across multiple adaptation models.

  • 3.3 MoA: A single-layer MoA uses Gemini-1.5-Flash to aggregate and synthesize outputs from multiple adaptation models.The adaptation models include Gemini, Gemma, GPT-4o-Mini, Meta-Llama, and Mistral variants.
  • 3.3 MoA: The MoA setup builds on the 5-shot setting used in previous steps.
  • 3.3 MoA: Finetuned Gemma-2-27B, Meta-Llama-3.1-8B, and Mistral-Nemo-Instruct-2407 models are integrated into MoA to assess finetuning’s impact on PLA performance.

3.4 Metrics for PLA task

PLA outputs are evaluated across relevance, readability, factuality, and LLM-judged quality dimensions using automatic metrics and human-like model assessment.

  • 3.4 Metrics for PLA task: The evaluation covers relevance, readability, factuality, and LLM-judged simplicity, accuracy, completeness, and brevity.
  • 3.4 Metrics for PLA task: BLEU, ROUGE, BERTScore, and SARI measure relevance against reference texts, with SARI specifically assessing appropriate simplification edits.
  • 3.4 Metrics for PLA task: FKGL, DCRS, and CLI assess accessibility through education level, word familiarity, and character and sentence length; lower scores indicate greater simplicity.
  • 3.4 Metrics for PLA task: AlignScore and SummaC evaluate factual consistency by detecting contradictions and aggregating sentence-level consistency evidence.
  • 3.4 Metrics for PLA task: Gemini-1.5-Flash serves as the LLM judge, using criteria refined for this study.The criteria are Simplicity, Accuracy, Completeness, and Brevity.

3.5 Metrics for term replacement task

The term replacement task evaluates classification accuracy with F1 while leaving generated replacements without automatic evaluation because no suitable reference exists.

  • 3.5 Metrics for term replacement task: Multilabel F1 measures accuracy for replacement type classification after difficult terms are identified.
  • 3.5 Metrics for term replacement task: Generated replacements are not automatically evaluated because prior work provides no suitable reference.

4 Experiments and Evaluations

Experiments evaluate prompting, finetuning, MoA, and term replacement on PLABA tasks using automated and human assessment. Few-shot semantic retrieval and finetuning improve several reported outcomes, while the shared-task system ranks competitively.

  • Few-shot ICL improves Relevance over zero-shot prompting, with semantically closest examples slightly outperforming random selection while Readability and Factuality change little.
  • Finetuning improves Relevance, Factuality, Readability, Accuracy, Completeness, and Brevity across model series, while judged Simplicity remains relatively consistent.
  • The term-replacement evaluation measures identification with F1 and classification with multilabel F1, using semantically closest few-shot examples.
  • Human evaluation scores Accuracy, Completeness, Simplicity, and Brevity on a -1, 0, or 1 scale, but some results are unavailable because too few judges participated.
  • Finetuned models ranked 4th on PLA with MoA, compared with 8th without finetuning, while the term-replacement system ranked 2nd for identification and overall text-generation average.

5 Discussion

Finetuning made readability more consistent across models, whereas non-finetuned models varied more and often over-simplified outputs. Evaluation remains constrained by metrics that do not fully capture the balance between simplicity and medical accuracy, and by reliance on the PLABA dataset.

  • 5.1 Readability: Finetuning made output readability more consistent across models and helped them adopt expert plain-language writing practices.Non-finetuned models varied more, while many generated overly simplified outputs that could sacrifice coherence and detail.
  • 5.1 Readability: The evaluation used human ratings of Simplicity, Accuracy, Completeness, Brevity, and overall Average scores, with some values unavailable because of limited judging.
  • 5.2 Auto Evaluation: Standard readability metrics such as FKGL and DCRS may miss the trade-off between simplifying medical content and preserving accuracy.
  • 5.3 Other Dataset: Training and testing only on PLABA may not represent the diversity of medical texts encountered in real-world applications.Additional datasets covering other medical domains and varied text structures would support broader evaluation of generalization.

6 Conclusion

The study finds that LLMs can simplify complex medical texts through few-shot prompting, finetuning, or Mixture-of-Agents methods. It also identifies general-purpose pretraining as a limitation and suggests healthcare-specific pretraining as future work.

  • 6 Conclusion: LLMs demonstrated potential for simplifying complex medical texts through plain-language adaptation using few-shot ICL, finetuning, or MoA.
  • 6 Conclusion: Few-shot prompting with the closest semantic similarity improved the relevance and readability of generated outputs.
  • 6 Conclusion: Finetuning enhanced PLA effectiveness by helping models adopt expert plain-language writing styles.
  • 6 Conclusion: The evaluated models were general-purpose LLMs rather than models pretrained on medical datasets, which may have limited domain-specific performance.The paper proposes healthcare-specific pretraining as a direction for improving PLA effectiveness and accuracy.
Loading 2609.17398v1…