PaperlayerView exact revision on arXiv

Source-linked AI summary

HerHealthGPT: An Evidence-Grounded Conversational AI System for Women's Reproductive Health Copy title

Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi, Hana Essam Sayed Ahmed Amrya, Mariam Mousa

arXiv:2610.33931v1cs.CL

TL;DR

Conversational AI may answer incomplete reproductive-health queries without collecting essential context or grounding guidance in verifiable evidence. HerHealthGPT combines retrieval, context-aware clarification, and source validation in an interactive system; retrieval-enabled concern classification and participant preferences were promising, while clinical validation remains necessary.

  • Problem

    Conversational systems may respond to incomplete reproductive-health queries without seeking essential context or grounding guidance in verifiable evidence.

    Source: §I · p. 1

  • Method

    HerHealthGPT combines domain-adapted language models, evidence retrieval, conversational history, targeted follow-up questions, and citation validation.

    Source: §I · p. 1

  • Results

    Concern-category accuracy rose from 54.1% with fine-tuning alone to 59.3% with retrieval, and 83.3% of participants preferred the retrieval-enabled configuration.

    Source: §I · p. 2, A. Component Evaluation · p. 4

  • Takeaways & Limitations

    The findings support evidence grounding as a promising component of context-aware reproductive-health information systems.

    Source: §V · p. 5

    The component evaluation used 90 source cases and heuristic labels, while the 30-person questionnaire and system-independent ratings limit comparative conclusions.

    Source: D. Discussion and Limitations · p. 5

Abstract

from arXiv · show

"Why is my period late?" Sometimes, the most useful response begins with another question. Yet conversational AI systems may respond to incomplete health queries without seeking essential context or grounding their guidance in verifiable evidence. This paper introduces HerHealthGPT, an evidence-grounded conversational AI system designed to address these challenges in women's reproductive health. HerHealthGPT integrates retrieval-augmented generation (RAG), context-aware clarification, and source validation within an interactive web interface. Rather than responding immediately to incomplete queries, the system maintains conversational context, asks targeted follow-up questions, checks for premature conclusions, and grounds its responses in retrieved evidence. We evaluate the AI components using 540 English messages derived from 90 reproductive-health cases and conduct a prototype questionnaire study with 30 adults. Evidence retrieval improves concern-category accuracy from 54.1% to 59.3% compared with fine-tuning alone. The clarification router achieves 85.6% recall and 86.2% specificity on constructed targets. In the user study, 83.3% of participants preferred the retrieval-enabled configuration, while 25 of 30 considered its follow-up questions relevant. These findings suggest that integrating evidence retrieval with interactive clarification offers a promising approach to more transparent, context-aware reproductive-health information systems, while further clinical validation remains necessary.

I. INTRODUCTION

Conversational health systems may answer incomplete reproductive-health queries without gathering essential context or grounding guidance in verifiable evidence. HerHealthGPT combines retrieval, domain adaptation, and structured clarification, with component and consumer evaluations reporting promising but limited results.

I. INTRODUCTION

  • Incomplete reproductive-health queries may omit symptom frequency, duration, or severity, while concerns can be expressed in clinical, everyday, or indirect language.The introduction argues that assistants should recognize uncertainty, ask relevant questions, and support responses with verifiable evidence.Source: §I · p. 1
  • HerHealthGPT combines domain-adapted language models, evidence retrieval, conversational context, targeted follow-up questions, and source-linked responses.Its workflow also checks for premature conclusions and validates citations.Source: §I · p. 1
  • 59.3% concern-category accuracy with retrieval, up from 54.1% with fine-tuning alone.The system also reports 85.6% recall for its exploratory clarification router and retrieval-enabled preference among 83.3% of 30 adults.Source: §I · p. 2

II. SYSTEM ARCHITECTURE

HerHealthGPT organizes reproductive-health assistance across four system layers. Its interface supports persistent, source-linked conversations and presents possible conditions as informational patterns rather than diagnoses.

II. SYSTEM ARCHITECTURE

  • The system integrates consumer interface, application services, model inference and retrieval, and persistent storage.This four-layer design supports follow-up questioning, evidence retrieval, and structured response generation.Source: §II · p. 2

A. Consumer Interface

  • The web interface supports desktop and mobile access, English and Arabic layouts, and persistent sessions that users can resume.Onboarding provides information about the system’s purpose and limitations.Source: A. Consumer Interface · p. 2
  • Users receive one follow-up question at a time with supporting sources, while conclusions provide source-linked informational patterns rather than diagnoses.Users can also request general information or submit out-of-scope questions without ending a session.Source: A. Consumer Interface · p. 2

B. Application and Evidence Services

Application services manage sessions and response handling, while a structured evidence store links reproductive-health features and conditions to traceable sources. Retrieval selects relevant sentence-level evidence subject to defined limits.

B. Application and Evidence Services

  • A FastAPI backend manages sessions, prompts, model requests, and response validation, while PostgreSQL stores conversation history, decisions, and citations.Generation uses Qwen3.5-9B; the application falls back to a hosted model if its adapted model is unavailable.Source: B. Application and Evidence Services · p. 2
  • The evidence store contains 39 askable features, 19 conditions, 146 evidence records, and 17 warning-sign rules drawn from five public sources.Each evidence record connects a feature and condition to a traceable source statement and evidence-strength label; an absent relationship is not evidence against a condition.Source: B. Application and Evidence Services · p. 2
  • BGE-M3 retrieval returns up to eight sentence-level candidates, reserving three positions for evidence statements and excluding cosine-similarity matches below 0.3.The product index excludes NHS content for licensing reasons and is separate from the 49-chunk evaluation corpus.Source: B. Application and Evidence Services · p. 2

C. Evidence-Grounded Conversational Control

HerHealthGPT combines conversation history and retrieved evidence to select structured response modes and action codes. Premature conclusions can prompt one revision, but the controls do not guarantee clinical correctness.

C. Evidence-Grounded Conversational Control

  • The system supports ask, conclude, inform, and decline response modes, with inform and decline not terminating the session.Conclusions use EMERGENCY, URGENT_SAME_DAY, ROUTINE_APPOINTMENT, and SELF_CARE action codes.Source: C. Evidence-Grounded Conversational… · pp. 2–3
  • Each turn uses conversation history and retrieved evidence to generate a structured decision, with a premature-conclusion check that may trigger one revision before validation and response delivery.The check considers whether a more severe warning sign is one answer away or whether fewer than three answers have been collected while an informative question remains.Source: Figure 2 · p. 3, C. Evidence-Grounded Conversational… · p. 3
  • Warning-sign definitions guide model-selected decisions rather than enforcing deterministic escalation, so the controls do not guarantee clinical correctness.Source: C. Evidence-Grounded Conversational… · p. 3

III. PROTOTYPE AND EVALUATION

The evaluation combines controlled component experiments with a consumer questionnaire, examining model adaptation, evidence retrieval, clarification, and user perceptions of the prototype.

A. Component Evaluation

  • The prototype evaluation uses 90 reproductive-health cases expressed as 540 English messages, including 90 constructed clarification-positive messages.The cases use six styles, and heuristic risk labels cover routine, same-day, and urgent messages.Source: A. Component Evaluation · p. 3
  • Six Qwen3.5-9B configurations compare zero-shot, RAG, fine-tuning, instruction, shuffled-evidence, and relevant-evidence variants.Adapted configurations share a QLoRA checkpoint; retrieval uses the top three passages from a 49-section NHS and NICHD corpus.Source: A. Component Evaluation · p. 3
  • Evaluation measures concern-category and risk accuracy, under-triage, and clarification recall and specificity.Category accuracy is label agreement, not factual correctness; full-sample and schema-compatible accuracy are reported because 72 labels fall outside the supported schema.Source: A. Component Evaluation · p. 3
  • Paired 95% confidence intervals use 10,000 source-case bootstrap replicates, preserving all six variations of each case.All configurations produced 540 parse-valid outputs under a common parser, with minor repairs for five outputs.Source: A. Component Evaluation · p. 3

B. Consumer Study

Thirty adults evaluated the web prototype across three configurations; the study materials show its onboarding, follow-up, and response interactions.

B. Consumer Study

  • Thirty adults participated; 25 evaluated the prototype in English and five in Arabic, and 25 had previously used chatbots for health information.Gender was not recorded.Source: B. Consumer Study · pp. 3–4
  • The prototype screens show one sourced follow-up question per turn, a source-linked routine-appointment conclusion, and a cited general answer without an urgency level.The caption characterizes these as interface examples, not clinical endorsements.Source: Figure 3 · p. 4
  • Participants interacted with a base model, a domain-adapted model, and a domain-adapted model with retrieval.These were configurations A, B, and C, respectively.Source: B. Consumer Study · p. 4

C. Questionnaire and Analysis

The questionnaire captures participants’ perceptions of the prototype and system preference, while the analysis limits comparisons between configurations.

C. Questionnaire and Analysis

  • The questionnaire assessed clarity, relevance, usefulness, hesitation, follow-up quality, perceived understanding, and confidence in sources.It also recorded system preference, intended actions, and qualitative feedback.Source: C. Questionnaire and Analysis · p. 4
  • Most questionnaire items used five-point scales, while perceived understanding retained its six-point scale.All 30 participants completed the ratings.Source: C. Questionnaire and Analysis · p. 4
  • The analysis reports medians, quartiles, and counts of ratings at least 4; system preference uses proportions and Wilson 95% confidence intervals.Source: C. Questionnaire and Analysis · p. 4
  • Ratings describe the overall prototype rather than individual systems; only the preference question compares configurations A, B, and C.Source: C. Questionnaire and Analysis · p. 4
  • Because exposure order was not recorded, the study performed no between-system significance tests.Arabic responses are treated as exploratory.Source: C. Questionnaire and Analysis · p. 4

IV. RESULTS AND DISCUSSION

  • The results section examines evidence retrieval, conversational-control tradeoffs, and user perceptions of the implemented prototype.Source: §IV · p. 4

A. Component Evaluation

Across component tests and the consumer questionnaire, retrieval improved concern classification, while urgency and clarification results remained qualified. Most participants preferred the retrieval-enabled system, but the study does not isolate retrieval’s effect.

A. Component Evaluation

  • 59.3% concern-category accuracy for FT+Real RAG exceeded FT’s 54.1% by 5.19 percentage points (95% CI [1.11, 9.26]) on 540 messages.The gain persisted on schema-compatible messages and after excluding one case with identified training-text overlap.Source: A. Component Evaluation · p. 4
  • Classification gains did not establish better urgency assessment: FT+Real RAG changed risk accuracy by −0.37 points (95% CI [−4.26, 3.33]) and under-triage by +4.02 points [−4.86, 13.10] versus FT.Only 18 urgent messages from three cases were available, and FT+Real RAG correctly predicted one as urgent.Source: A. Component Evaluation · p. 4

B. Exploratory Clarification Routing

  • The exploratory router achieved 85.6% recall and 86.2% specificity, but its metrics were inherited from ZS after benchmark inspection and do not validate the deployed conversational control loop.The live wrapper was smoke-tested on six variations of one case.Source: B. Exploratory Clarification Routing · p. 4

C. Consumer Evaluation

  • 25 of 30 participants preferred System C, which combines domain adaptation and retrieval; they cited follow-up questions, perceived reliability, response speed, and supporting sources.The questionnaire did not collect separate ratings for each configuration or record exposure order, so it indicates favorable perceptions rather than an isolated retrieval effect.Source: C. Consumer Evaluation · p. 5
  • Participants most often selected online searching or home monitoring as intended next actions (10 each), while eight expressed hesitation about relying on sources and thirteen disagreed.These were self-reported choices, not verified clinical decisions.Source: C. Consumer Evaluation · p. 5

D. Discussion and Limitations

The discussion distinguishes improved concern classification from evidence of better urgency assessment or clinical reliability. It also identifies limits in the component evaluation, clarification results, and consumer study.

D. Discussion and Limitations

  • Improved concern classification does not establish better urgency assessment or clinical reliability.Clarification routing shows promise but may introduce unnecessary questions.Source: D. Discussion and Limitations · p. 5
  • The component evaluation is limited by 90 source cases, heuristic labels, schema mismatches, and potential training-data overlap.Further work should evaluate end-to-end conversational reliability, clinical appropriateness, latency, and deployment performance.Source: D. Discussion and Limitations · p. 5
  • The questionnaire’s small sample (n = 30) and system-independent ratings limit comparative conclusions; five Arabic responses cannot establish multilingual performance.Source: D. Discussion and Limitations · p. 5

V. CONCLUSION

HerHealthGPT integrates domain-adapted language models, retrieval, and structured conversational control for reproductive-health information. Its results include improved concern classification and strong preference for the retrieval-enabled configuration, while underscoring that classification gains do not guarantee clinically appropriate guidance.

V. CONCLUSION

  • HerHealthGPT integrates domain-adapted language models, retrieval, and structured conversational control for women’s reproductive health.Source: §V · p. 5
  • Concern classification reached 59.3% accuracy, a statistically supported 5.19 percentage-point gain over fine-tuning alone.Source: §V · p. 5
  • The retrieval-enabled configuration was preferred by 83.3% of participants, but improved classification does not guarantee clinically appropriate guidance.The paper presents the system as a practical foundation and identifies clinician-validated evaluation, safer conversational decision-making, and robust multilingual support as future priorities.Source: §V · p. 5

Related papers

Loading 2610.33931v1…